Streaming methods and systems

• US 20030195977A1
• Filed: 04/11/2002
• Published: 10/16/2003
• Est. Priority Date: 04/11/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Hong-Jiang, Qi, Wei, Liu, Tianming
• Granted Patent: US 7,483,487 B2
• Time in Patent Office: Days
• US Class Current: 709/231
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 65/752 adapting media to network capabilities
• CPC Class Codes: H04L 65/762 at the source reformatting of additional data in video distribution servers H04N21/2355
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/152 by measuring the fullness of the transmission buffer
• CPC Class Codes: H04N 19/159 Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/34 Scalability techniques involving progressive bit-plane based encoding of the enhancement layer, e.g. fine granular scalability [FGS]
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/87 involving scene cut or scene change detection in combination with video compression
• CPC Class Codes: H04N 21/234327 by decomposing into layers, e.g. base layer and one or more enhancement layers
• CPC Class Codes: H04N 21/2401 Monitoring of the client buffer
• CPC Class Codes: H04N 21/2402 Monitoring of the downstream path of the transmission network, e.g. bandwidth available traffic monitoring in data switching networks H04L43/00; monitoring data switching networks utilization H04L43/0876
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/6373 for rate control , e.g. request to the server to modify its transmission rate flow control in packet networks H04L47/10

Abstract

Various embodiments provide methods and systems for streaming data that can facilitate streaming during bandwidth fluctuations in a manner that can enhance the user experience. In one aspect, a forward-shifting technique is utilized to buffer data that is to be streamed, e.g. an enhancement layer in a FGS stream. Various techniques can drop layers actively when bandwidth is constant. The saved bandwidth can then be used to pre-stream enhancement layer portions. In another aspect, a content-aware decision can be made as to how to drop enhancement layers when bandwidth decreases. During periods of decreasing bandwidth, if a video segment does not contain important content, the enhancement layers will be dropped to keep the forward-shifting of the enhancement layer unchanged. If the enhancement layer does contain important content, it will be transmitted later when bandwidth increases.

Claims

1. A method comprising:

providing a streaming server configured to stream data; and

at the streaming server, dropping at least portions of an enhancement layer sufficient to forward-shift following portions of the enhancement layer by an amount.

2. The method of claim 1 further comprising:

determining whether portions of the enhancement layer are important; and

making a decision to drop enhancement layer portions based, at least in part, on whether the portions are important.

3. The method of claim 1 further comprising:

determining a client side buffer size; and

making a decision to drop enhancement layer portions based, at least in part, on the client side buffer size.

4. The method of claim 1, wherein the enhancement layer comprises individual blocks having block sizes, and said amount by which said other portions are shifted comprises at least one block size.

5. The method of claim 4, wherein individual block sizes contain the same amount of bits.

6. The method of claim 1, wherein dropping said portions of the enhancement layer comprise dropping portions that are less important than portions that are not dropped.

7. The method of claim 1, wherein the enhancement layer comprises part of an FGS stream.

8. The method of claim 1 further comprising if the enhancement layer contains important content and bandwidth is insufficient, delaying transmission of portions of the enhancement layer that contain the important content.

9. The method of claim 1, wherein said act of dropping comprises dropping at least one enhancement layer.

10. The method of claim 1, wherein said act of dropping comprises dropping at least one frame.

11. The method of claim 1, wherein said act of dropping comprises dropping at least one block.

12. One or more computer-readable media having computer-readable instructions thereon which, when executed by one or more processors, cause the one or more processors to implement the method of claim 1.

13. A method comprising:

determining an importance of content that is to be streamed to a client by a steaming server; and

making a decision to forward-shift following portions of the content to the client based on the determined importance of the content.

14. The method of claim 13, wherein said act of making a decision comprises deciding, when network bandwidth is more available, to more actively forward shift content that is determined to be important than content that is determined to be not as important.

15. The method of claim 13, wherein said act of making a decision comprises deciding, when network bandwidth is less available, to less actively forward shift content that is determined to be less important.

16. The method of claim 13, wherein said act of making a decision comprises:

deciding, when network bandwidth is more available, to more actively forward shift content that is determined to be important than content that is determined to be not as important; and

deciding, when network bandwidth is less available, to less actively forward shift content that is determined to be less important.

17. A method comprising:

first dropping at least portions of an enhancement layer sufficient to forward-shift following portions of the enhancement layer by an amount;

determining whether a decrease in available bandwidth has occurred;

responsive to said determining, determining whether the enhancement layer contains important content, and;

if the enhancement layer does not contain important content, second dropping additional portions of the enhancement layer in an attempt to preserve a forward-shifted enhancement layer, otherwise,

if the enhancement layer does contain important content, delaying transmission of portions of the enhancement layer so that the delayed portions can be transmitted when the available bandwidth increases.

18. The method of claim 17, wherein said first dropping occurs in the absence of a decrease in available bandwidth.

19. The method of claim 17, wherein the enhancement layer comprises individual blocks having block sizes and said amount comprises at least one block size.

20. The method of claim 19, wherein individual block sizes contain the same amount of bits.

21. The method of claim 17, wherein the enhancement layer comprises part of an FGS stream.

22. The method of claim 17, wherein said act of first dropping occurs prior to streaming any enhancement layer portions.

23. The method of claim 17 further comprising:

determining that the available bandwidth has increased;

responsive to determining that the available bandwidth has increased, transmitting the delayed portions of the enhancement layer; and

after said transmitting of the delayed portions, dropping additional portions of the enhancement layer sufficient to forward shift following portions of the enhancement layer by an amount.

24. The method of claim 17, wherein said acts of dropping comprise dropping one or more enhancement layers.

25. The method of claim 17, wherein said acts of dropping comprise dropping one or more frames.

26. The method of claim 17, wherein said acts of dropping comprise dropping one or more blocks.

27. One or more computer-readable media having computer-readable instructions thereon which, when executed by one or more processors, cause the one or more processors to implement the method of claim 17.

28. A method comprising:

actively dropping portions of an enhancement layer when available bandwidth is constant to effectively forward shift other portions of the enhancement layer;

analyzing content of the enhancement layer to determine its relative importance; and

if content is determined to not be important, dropping associated portions of the enhancement layer, otherwise, delaying transmission of associated portions of the enhancement layer.

29. The method of claim 28, wherein said act of analyzing is performed responsive to a decrease in available bandwidth.

30. The method of claim 28, wherein said enhancement layer comprises part of an FGS stream.

31. The method of claim 28, wherein said dropping of the enhancement layer portions comprises dropping one or more enhancement layers.

32. The method of claim 28, wherein said dropping of the enhancement layer portions comprises dropping one or more frames.

33. The method of claim 28, wherein said dropping of the enhancement layer portions comprises dropping one or more blocks.

34. One or more computer-readable media having computer-readable instructions thereon which, when executed by one or more processors, cause the one or more processors to implement the method of claim 28.

35. A streaming server comprising:

one or more processors;

memory; and

software code embodiment in the memory which, when executed by the one or more processors, cause the one or more processors to drop at least portions of an enhancement layer sufficient to forward-shift following portions of the enhancement layer by an amount.

36. The streaming server of claim 35, wherein the software code causes the one or more processors to:

determine whether portions of the enhancement layer are important; and

make a decision to drop enhancement layer portions based, at least in part, on whether the portions are important.

37. The streaming server of claim 35, wherein the software code causes the one or more processors to:

determine a client side buffer size; and

make a decision to drop enhancement layer portions based, at least in part, on the client side buffer size.

38. The streaming server of claim 35, wherein the software code causes the one or more processors to drop portions of the enhancement layer that are less important than portions that are not dropped.

39. A streaming server comprising:

one or more processors;

memory;

software code embodiment in the memory which, when executed by the one or more processors, cause the one or more processors to;

first drop at least portions of an enhancement layer sufficient to forward-shift following portions of the enhancement layer by an amount;

determine whether a decrease in available bandwidth has occurred;

responsive to determining that a decrease in available bandwidth has occurred, determine whether the enhancement layer contains important content, and;

if the enhancement layer does not contain important content, second drop additional portions of the enhancement layer in an attempt to preserve a forward-shifted enhancement layer, otherwise,

if the enhancement layer does contain important content, delay transmission of portions of the enhancement layer so that the delayed portions can be transmitted when the available bandwidth increases.

40. The streaming server of claim 39, wherein the software code causes the one or more processors to first drop said portions in the absence of a decrease in available bandwidth.

41. The streaming server of claim 39, wherein the software code causes the one or more processors to first drop said portions prior to streaming any enhancement layer portions.

42. The streaming server of claim 39, wherein the software code causes the one or more processors to:

determine that the available bandwidth has increased;

responsive to determining that the available bandwidth has increased, transmit the delayed portions of the enhancement layer; and

after transmitting the delayed portions, drop additional portions of the enhancement layer sufficient to forward shift other portions of the enhancement layer by an amount.

43. The streaming server of claim 39, wherein the software code causes the one or more processors to drop one or more enhancement layers.

44. The streaming server of claim 39, wherein the software code causes the one or more processors to drop one or more frames.

45. The streaming server of claim 39, wherein the software code causes the one or more processors to drop one or more blocks.

46. One or more computer-readable media having computer-readable instructions thereon which, when executed by one or more processors, cause the processors to operate a streaming server by dropping at least portions of an enhancement layer comprising part of an FGS stream sufficient to forward-shift other portions of the enhancement layer by an amount.

47. The one or more computer-readable media of claim 46, wherein the computer-readable instructions cause the processors to drop portions that are less important than other portions.

48. The one or more computer-readable media of claim 46, wherein the computer-readable instructions cause the processors to determine whether the enhancement layer contains important content;

and

if the enhancement layer does not contain important content, drop additional portions of the enhancement layer in an attempt to preserve a forward-shifted enhancement layer; and

if the enhancement layer contains important content, delay transmission of portions of the enhancement layer that contain the important content.

49. The one or more computer-readable media of claim 46, wherein the computer-readable instructions cause the processors to drop one or more enhancement layers.

50. The one or more computer-readable media of claim 46, wherein the computer-readable instructions cause the processors to drop one or more frames.

51. The one or more computer-readable media of claim 46, wherein the computer-readable instructions cause the processors to drop one or more block.

52. One or more computer-readable media having computer-readable instructions thereon which, when executed by one or more processors, cause the processors to:

first drop at least portions of an enhancement layer comprising part of an FGS stream sufficient to forward-shift other portions of the enhancement layer by an amount;

determine whether a decrease in available bandwidth has occurred;

responsive to determining that a decrease in available bandwidth has occurred, determine whether the enhancement layer contains important content, and;

if the enhancement layer does not contain important content, second drop additional portions of the enhancement layer in an attempt to preserve a forward-shifted enhancement layer, otherwise,

if the enhancement layer does contain important content, delay transmission of portions of the enhancement layer so that the delayed portions can be transmitted when the available bandwidth increases;

determine that the available bandwidth has increased

responsive to determining that the available bandwidth has increased, transmit the delayed portions of the enhancement layer; and

after transmitting the delayed portions, drop additional portions of the enhancement layer sufficient to forward shift other portions of the enhancement layer by an amount.

53. The one or more computer-readable media of claim 52, wherein the computer-readable instructions cause the processors to first drop said enhancement layer portions in the absence of a decrease in available bandwidth.

54. The one or more computer-readable media of claim 52, wherein the computer-readable instructions cause the processors to first drop said enhancement layer portions prior to streaming any enhancement layer portions.

55. One or more computer-readable media having computer-readable instructions thereon which, when executed by one or more processors, cause the processors to:

actively drop portions of an enhancement layer comprising part of an FGS stream when available bandwidth is constant to effectively forward shift following portions of the enhancement layer;

analyze content of the enhancement layer to determine its relative importance; and

if content is determined to not be important, drop associated portions of the enhancement layer, otherwise, delay transmission of associated portions of the enhancement layer.

56. The one or more computer-readable media of claim 55, wherein the computer-readable instructions cause the processors to analyze said content responsive to a decrease in available bandwidth.

57. A method comprising:

providing a streaming server configured to stream data to at least one client, said data comprising one or more enhancement layers;

determining whether portions of the one or more enhancement layers are important by taking into account motion that is embodied in said portions;

making a decision to drop enhancement layer portions based, at least in part, on whether the portions are important.

58. The method of claim 57, wherein said act of determining comprises:

computing perceived motion energy (PME) values for multiple B frames of a video stream;

representing the video stream as a PME value sequence;

segmenting the PME value sequence into individual segments, each segment corresponding to multiple frames;

within each segment, assigning one or more priority levels; and

wherein said act of making a decision comprises dropping frames having lower priorities when available bandwidth decreases.

59. The method of claim 58, wherein said computing comprises taking the product of an average of motion vector magnitudes within a frame and a percentage of dominant motion direction.

60. The method of claim 58, wherein the PME values reflect a degree of perceived motion judder if a frame is dropped.

61. The method of claim 58, wherein the multiple frames comprise B frames.

62. The method of claim 58, wherein said segmenting comprises using a model to detect patterns in the PME value sequence so that the detected patterns can be segmented into individual segments.

63. The method of claim 62, wherein said model comprises a triangle model.

64. A method comprising:

computing perceived motion energy (PME) values for multiple frames of a video stream;

representing the video stream as a PME value sequence;

segmenting the PME value sequence into individual segments, each segment corresponding to multiple frames;

within each segment, assigning one or more priority levels; and

dropping frames having lower priorities when available bandwidth decreases.

65. The method of claim 64, wherein said computing comprises taking the product of an average of motion vector magnitudes within a frame and a percentage of dominant motion direction.

66. The method of claim 64, wherein the PME values reflect a degree of perceived motion judder if a frame is dropped.

67. The method of claim 64, wherein the multiple frames comprise B frames.

68. The method of claim 64, wherein said segmenting comprises using a model to detect patterns in the PME value sequence so that the detected patterns can be segmented into individual segments.

69. The method of claim 68, wherein said model comprises a triangle model.

70. The method of claim 69, wherein left bottom vertices of the model represent starting points of individual segments, right bottom vertices of the model represent end points of individual segments, and top vertices of the model represent maximum PME values of individual segments.

71. The method of claim 64, wherein said segmenting comprises, prior to using the model, splitting one or more portions of the PME value sequence that correspond to continuous motion by defining splitting boundaries relative to the PME value sequence.

72. The method of claim 64, wherein said assigning of the priority levels takes place by considering degraded quality if frames are dropped.

73. The method of claim 64 further comprising dropping frames with lower priority levels when bandwidth is constant to forward shift frames that are not dropped.

74. The method of claim 64, wherein said assigning of the priority levels comprises assigning individual B frame portions a class in accordance with the peak PME value of its associated segment, wherein any one class is associated with one or more models that provide a mapping between classes and individual priority levels.

75. A method comprising:

providing one or more models that associate perceived motion energy (PME) values for video segments with individual classes, each class being associated with a score that pertains to human-perceived impairment caused by frame rate decreases;

assigning individual video segments a class in accordance with a segment'"'"'s peak PME value;

mapping the assigned class to an individual priority level; and

using the priority levels to ascertain one or more frames within a video segment to drop.

76. The method of claim 75, wherein providing one or more models comprises providing multiple models each of which being associated with a different frame rate.

77. The method of claim 75, wherein said using comprises dropping frames having lower priority levels than other frames having higher priorities.

78. The method of claim 75, wherein said using comprises dropping frames having lower priority levels than other frames having higher priorities when available bandwidth is stable.

79. The method of claim 75, wherein said one or more frames comprise B-frames.

80. A system comprising:

a rate controller comprising;

a state machine for implementing perceptual rate adaptation through mode and state transitions;

a virtual buffer model communicatively coupled with and providing feedback to the state machine for describing dynamical buffer filling and draining processes; and

a bandwidth allocation module communicatively coupled with the state machine for allocating bandwidth to frames given the state and mode of the state machine.

81. The system of claim 80, wherein said frames comprise B frames.

82. The system of claim 80, wherein the virtual buffer model comprises a frame FIFO queue.

83. The system of claim 80, wherein the virtual buffer model comprises:

an input for a sending frame rate associated with a sliding window size;

an output associated with a constant frame consumption rate; and

a feedback output provided to the state machine and associated with a number of buffered frames after a given time slot.

84. The system of claim 80, wherein state machine modes define protected priority levels associated with frames that are to be protected.

85. The system of claim 84, wherein priority levels that are higher or equal to a current mode are protected.

86. The system of claim 84, wherein the state machine is configured to transit modes if bandwidth or stream bit rate changes.

87. The system of claim 80, wherein the state machine states determine a current window size that is associated with a sending frame rate.

88. The system of claim 87, wherein within each mode, state transits are configured to absorb short-term bandwidth or stream bit-rate variations.

89. The system of claim 80, wherein the state machine comprises:

a first state associated with a state in which available bandwidth is not fluctuating;

a second state in which the available bandwidth is insufficient; and

a third state in bandwidth recovers from the second state.

90. The system of claim 89, wherein:

the first state attempts to maintain a window size associated with a sending frame rate;

the second state decreases the window size and uses a client buffer for additional bandwidth; and

the third state increases the window size by actively dropping unprotected frames.

91. The system of claim 90, wherein the third state begins to fill a client buffer in anticipation of a bandwidth decrease.

92. A method comprising:

providing a state machine for implementing perceptual rate adaptation through mode and state transitions, wherein state machine modes define protected priority levels associated with frames that are to be protected, the state machine being configured to transit modes if bandwidth or stream bit rate changes in the sense of long-term, the state machine comprising a first state associated with a state in which available bandwidth is not fluctuating and in which an attempt is made to maintain a window size associated with a sending frame rate, a second state in which the available bandwidth is insufficient and in which the window size is decreased and a client buffer is used for additional bandwidth, and a third state in bandwidth recovers from the second state and in which the window size is increased by actively dropping unprotected frames;

providing a virtual buffer model communicatively coupled with and providing feedback to the state machine for describing dynamical buffer filling and draining processes, the virtual buffer model comprising an input for the sending frame rate associated with the window size, an output associated with a constant frame consumption rate, and a feedback output provided to the state machine and associated with a number of buffered frames after a given time slot; and

providing a bandwidth allocation module communicatively coupled with the state machine for allocating bandwidth to frames given the state and mode of the state machine.