Streaming Methods and Systems

• US 20090185618A1
• Filed: 01/20/2009
• Published: 07/23/2009
• Est. Priority Date: 04/11/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Hong-Jiang, Qi, Wei, Liu, Tianming
• Granted Patent: US 8,094,719 B2
• Time in Patent Office: Days
• US Class Current: 375/240.10
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 65/752 adapting media to network capabilities
• CPC Class Codes: H04L 65/762 at the source reformatting of additional data in video distribution servers H04N21/2355
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/152 by measuring the fullness of the transmission buffer
• CPC Class Codes: H04N 19/159 Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/34 Scalability techniques involving progressive bit-plane based encoding of the enhancement layer, e.g. fine granular scalability [FGS]
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/87 involving scene cut or scene change detection in combination with video compression
• CPC Class Codes: H04N 21/234327 by decomposing into layers, e.g. base layer and one or more enhancement layers
• CPC Class Codes: H04N 21/2401 Monitoring of the client buffer
• CPC Class Codes: H04N 21/2402 Monitoring of the downstream path of the transmission network, e.g. bandwidth available traffic monitoring in data switching networks H04L43/00; monitoring data switching networks utilization H04L43/0876
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/6373 for rate control , e.g. request to the server to modify its transmission rate flow control in packet networks H04L47/10

Abstract

Various embodiments provide methods and systems for streaming data that can facilitate streaming during bandwidth fluctuations in a manner that can enhance the user experience. In one aspect, a forward-shifting technique is utilized to buffer data that is to be streamed, e.g. an enhancement layer in a FGS stream. Various techniques can drop layers actively when bandwidth is constant. The saved bandwidth can then be used to pre-stream enhancement layer portions. In another aspect, a content-aware decision can be made as to how to drop enhancement layers when bandwidth decreases. During periods of decreasing bandwidth, if a video segment does not contain important content, the enhancement layers will be dropped to keep the forward-shifting of the enhancement layer unchanged. If the enhancement layer does contain important content, it will be transmitted later when bandwidth increases.

Claims

1. A video streaming system comprising:

a processor;

a memory; and

a rate controller maintained in the memory and executed on the processor, the rate controller comprising;

a state machine for implementing perceptual rate adaptation based on mode transitions from a first mode to a second mode, and state transitions from a first state to a second state;

a virtual buffer module communicatively coupled with, and

providing feedback to, the state machine for describing dynamic buffer filling and draining processes; and

a bandwidth allocation module communicatively coupled with the state machine for allocating bandwidth to frames based on a current state and a current mode of the state machine.

2. The system of claim 1, wherein the frames comprise B frames.

3. The system of claim 1, wherein the virtual buffer module comprises a first-in-first out frame queue.

4. The system of claim 1, wherein the virtual buffer module is configured to:

receive an input that defines a sending frame rate associated with a sliding window size;

generate an output associated with a constant frame consumption rate; and

provide to the state machine, a feedback output that is associated with a number of buffered frames after a given time slot.

5. The system of claim 1, wherein state machine modes define protected priority levels associated with frames that are to be protected.

6. The system of claim 5, wherein priority levels that are higher than, or equal to, a priority level defined by a current mode are protected.

7. The system of claim 5, wherein the state machine is configured to transition between modes when bandwidth changes.

8. The system of claim 5, wherein the state machine is configured to transition between modes when stream bit rate changes.

9. The system of claim 1, wherein states of the state machine determine a current window size whereby the current window size is associated with a sending frame rate.

10. The system of claim 8, wherein within each mode, state transitions are configured to absorb short-term bandwidth variations.

11. The system of claim 8, wherein within each mode, state transitions are configured to absorb short-term stream bit-rate variations.

12. The system of claim 1, wherein the state machine comprises:

a first state in which available bandwidth is not fluctuating;

a second state in which the available bandwidth is insufficient; and

a third state in which available bandwidth is sufficient after having been insufficient

13. The system of claim 12, wherein the state machine, when in

the first state, attempts to maintain a window size associated with a sending frame rate.

14. The system of claim 12, wherein the state machine, when in

the second state, decreases the window size and uses a client buffer for additional bandwidth.

15. The system of claim 12, wherein the state machine, when in

the third state increases the window size by actively dropping unprotected frames.

16. The system of claim 13, wherein the state machine, when in the third state, begins to fill a client buffer in anticipation of a bandwidth decrease.

17. A method for providing a state machine for implementing perceptual video stream rate adaptation through mode and state transitions, wherein state machine modes define protected priority levels associated with frames of the video stream that are to be protected, the method comprising:

Transitioning the state machine from one mode to another mode when bandwidth changes to a second non-fluctuating level,

the state machine maintaining a first state wherein the available bandwidth is non-fluctuating and in which an attempt is made to maintain a window size associated with a sending frame rate,

the state machine maintaining a second state wherein the available bandwidth is insufficient and in which the window size is decreased and a client buffer is used to compensate for lack of additional bandwidth, and

the state machine maintaining a third state wherein bandwidth recovers from the second state and in which the window size is increased by actively dropping unprotected frames;

providing feedback the to the state machine for describing dynamical buffer filling and draining processes from a virtual buffer module communicatively coupled with the state machine;

the virtual buffer module facilitating;

an input interface facilitating receiving the sending frame rate associated with the window size,

an output interface facilitating providing constant frame consumption rate data, and

a feedback interface providing data associated with a number of buffered frames after a given time slot; and

providing a bandwidth allocation module communicatively coupled with the state machine for allocating bandwidth to frames given the state and mode of the state machine.

18. A computer readable storage media encoded with instructions that when executed by a processor implement the method recited in claim 17.

19. A computer readable storage media encoded with instructions that when executed by a processor execute a method, the method comprising:

a state machine maintaining a first state wherein the available bandwidth is non-fluctuating and in which an attempt is made to maintain a window size associated with a sending frame rate,

the state machine maintaining a second state wherein the available bandwidth is insufficient and in which the window size is decreased and a client buffer is used to compensate for lack of additional bandwidth, and

the state machine maintaining a third state wherein bandwidth recovers from the second state and in which the window size is increased by actively dropping unprotected frames; and

providing a bandwidth allocation module communicatively coupled with the state machine for allocating bandwidth to frames given the state and mode of the state machine.

20. The computer readable storage medium as recited in claim 19 further encoded such that when executed by a processor execute the method of claim 19 the method further comprising:

providing feedback the to the state machine for describing dynamical buffer filling and draining processes from a virtual buffer module communicatively coupled with the state machine;

the virtual buffer module facilitating;

an input interface facilitating receiving the sending frame rate associated with the window size,

an output interface facilitating providing constant frame consumption rate data, and

a feedback interface providing data associated with a number of buffered frames after a given time slot.