Streaming methods and systems

• US 7,483,487 B2
• Filed: 04/11/2002
• Issued: 01/27/2009
• Est. Priority Date: 04/11/2002
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Qi, Wei, Liu, Tianming, Zhang, Hong-Jiang
• Primary Examiner(s): DIEP, NHON THANH
• Application Number:
  US10/121,333
• Publication Number:
  US 20030195977A1
• Time in Patent Office: 2,483 Days
• Field of Search: None
• US Class Current: 375/240.11
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 65/752 adapting media to network capabilities
• CPC Class Codes: H04L 65/762 at the source reformatting of additional data in video distribution servers H04N21/2355
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/152 by measuring the fullness of the transmission buffer
• CPC Class Codes: H04N 19/159 Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/34 Scalability techniques involving progressive bit-plane based encoding of the enhancement layer, e.g. fine granular scalability [FGS]
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/87 involving scene cut or scene change detection in combination with video compression
• CPC Class Codes: H04N 21/234327 by decomposing into layers, e.g. base layer and one or more enhancement layers
• CPC Class Codes: H04N 21/2401 Monitoring of the client buffer
• CPC Class Codes: H04N 21/2402 Monitoring of the downstream path of the transmission network, e.g. bandwidth available traffic monitoring in data switching networks H04L43/00; monitoring data switching networks utilization H04L43/0876
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/6373 for rate control , e.g. request to the server to modify its transmission rate flow control in packet networks H04L47/10

Abstract

Various embodiments provide methods and systems for streaming data that can facilitate streaming during bandwidth fluctuations in a manner that can enhance the user experience. In one aspect, a forward-shifting technique is utilized to buffer data that is to be streamed, e.g. an enhancement layer in a FGS stream. Various techniques can drop layers actively when bandwidth is constant. The saved bandwidth can then be used to pre-stream enhancement layer portions. In another aspect, a content-aware decision can be made as to how to drop enhancement layers when bandwidth decreases. During periods of decreasing bandwidth, if a video segment does not contain important content, the enhancement layers will be dropped to keep the forward-shifting of the enhancement layer unchanged. If the enhancement layer does contain important content, it will be transmitted later when bandwidth increases.

Claims

1. A method comprising:

providing a streaming server configured to stream data to at least one client, said data comprising one or more enhancement layers;

determining whether portions of the one or more enhancement layers are important by taking into account motion that is embodied in said portions, the determining comprising;

computing perceived motion energy (PME) values for multiple B frames of a video stream;

taking the product of an average of motion vector magnitudes within a frame and a percentage of dominant motion direction;

representing the video stream as a PME value sequence;

segmenting the PME value sequence into individual segments, each segment corresponding to multiple frames;

within each segment, assigning one or more priority levels; and

making a decision to drop enhancement layer portions based, at least in part, on whether the portions are important dropping frames having lower priorities when available bandwidth decreases.

2. The method of claim 1, wherein the PME values reflect a degree of perceived motion judder if a frame is dropped.

3. The method of claim 1, wherein the multiple frames comprise B frames.

4. The method of claim 1, wherein said segmenting comprises using a model to detect patterns in the PME value sequence so that the detected patterns can be segmented into individual segments.

5. The method of claim 4, wherein said model comprises a triangle model.

6. A method comprising:

computing perceived motion energy (PME) values for multiple frames of a video stream by taking the product of an average of motion vector magnitudes within a frame and a percentage of dominant motion direction;

representing the video stream as a PME value sequence;

segmenting the PME value sequence into individual segments, each segment corresponding to multiple frames;

within each segment, assigning one or more priority levels; and

dropping frames having lower priorities when available bandwidth decreases.

7. The method of claim 6, wherein the PME values reflect a degree of perceived motion judder if a frame is dropped.

8. The method of claim 6, wherein the multiple frames comprise B frames.

9. The method of claim 6, wherein said segmenting comprises using a model to detect patterns in the PME value sequence so that the detected patterns can be segmented into individual segments.

10. The method of claim 9, wherein said model comprises a triangle model.

11. The method of claim 10, wherein left bottom vertices of the model represent starting points of individual segments, right bottom vertices of the model represent end points of individual segments, and top vertices of the model represent maximum PME values of individual segments.

12. The method of claim 6, wherein said segmenting comprises, prior to using the model, splitting one or more portions of the PME value sequence that correspond to continuous motion by defining splitting boundaries relative to the PME value sequence.

13. The method of claim 6, wherein said assigning of the priority levels takes place by considering degraded quality if frames are dropped.

14. The method of claim 6 further comprising dropping frames with lower priority levels when bandwidth is constant to forward shift frames that are not dropped.

15. The method of claim 6, wherein said assigning of the priority levels comprises assigning individual B frame portions a class in accordance with the peak PME value of its associated segment, wherein any one class is associated with one or more models that provide a mapping between classes and individual priority levels.