Fine-grained client-side control of scalable media delivery

• US 8,265,140 B2
• Filed: 09/30/2008
• Issued: 09/11/2012
• Est. Priority Date: 09/30/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Mehrotra, Sanjeev
• Primary Examiner(s): Chai, Longbit
• Application Number:
  US12/242,524
• Publication Number:
  US 20100080290A1
• Time in Patent Office: 1,442 Days
• Field of Search: 375/240.02
• US Class Current: 375/240.02
• CPC Class Codes: H04N 19/115 Selection of the code volume for a coding unit prior to coding
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/147 according to rate distortion criteria rate-distortion as a criterion for motion estimation H04N19/567
• CPC Class Codes: H04N 19/164 Feedback from the receiver or from the transmission channel
• CPC Class Codes: H04N 19/187 the unit being a scalable video layer
• CPC Class Codes: H04N 19/19 using optimisation based on Lagrange multipliers
• CPC Class Codes: H04N 19/30 using hierarchical techniques, e.g. scalability H04N19/63 takes precedence
• CPC Class Codes: H04N 19/34 Scalability techniques involving progressive bit-plane based encoding of the enhancement layer, e.g. fine granular scalability [FGS]
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/61 in combination with predictive coding

Abstract

Techniques and tools for adjusting quality and bit rate of multiple chunks of media delivered over a network are described. For example, each of the multiple chunks is encoded as multiple layers (e.g., a base layer and multiple embedded residual layers) for fine-grained scalability at different rate/quality points. A server stores the encoded data for the layers of chunks as well as curve information that parameterizes rate-distortion curves for the chunks. The server sends the curve information to a client. For the multiple chunks, the client uses the curve information to determine rate-distortion preferences for the respective chunks, then sends feedback indicating the rate-distortion preferences to the server. For each of the multiple chunks, the server, based at least in part upon the feedback, selects one or more scalable layers of the chunk to deliver to the client.

Claims

1. A method of facilitating client-side control of delivery of media data organized as plural chunks to a media client across a network, a given chunk of the plural chunks being encoded as plural scalable layers for different quality levels with fine-grained scalability, the method comprising:

with a computing system that implements a media server, sending, for receipt by the media client across the network, curve information, wherein the curve information at least in part indicates distortion as a function of download time for the given chunk to facilitate determination of a rate-distortion preference for the given chunk, by the media client, that includes;

determining a playback time constraint for a segment of one or more chunks of the plural chunks, the segment including the given chunk; and

for each of the one or more chunks in the segment, selecting a download time so as to reduce overall distortion among the one or more chunks in the segment while satisfying the playback time constraint for the segment; and

for the given chunk of the plural chunks;

receiving feedback from the media client that indicates the rate-distortion preference for the given chunk;

with the computing system that implements the media server, based at least in part upon the rate-distortion preference, selecting one or more scalable layers from among the plural scalable layers of the given chunk; and

sending the selected one or more scalable layers of the given chunk.

2. The method of claim 1 wherein each of the plural chunks is encoded without dependencies on any other chunk of the plural chunks, and wherein the method further comprises sending, for receipt by the media client across the network, a stream header for the media data organized as plural chunks.

3. The method of claim 1 wherein, for the given chunk of the plural chunks:

the plural scalable layers include a base layer and plural residual layers;

the base layer is always selected and sent for the given chunk; and

zero or more of the plural residual layers are selected and sent for the given chunk, depending on the rate-distortion preference.

4. The method of claim 1 wherein, for the given chunk of the plural chunks, the plural scalable layers include one or more residual embedded layers resulting from bit plane encoding of the given chunk, each of the one or more residual embedded layers representing a bit plane of transform coefficients of the given chunk.

5. The method of claim 1 wherein, for the given chunk of the plural chunks, the curve information includes rate-distortion points that parameterize a rate-distortion curve for the given chunk.

6. The method of claim 5 wherein, for each of the plural scalable layers of the given chunk, the curve information includes rate information that indicates a bit count for the scalable layer.

7. The method of claim 1 wherein, for the given chunk of the plural chunks, a set of parameters in the curve information for the chunk indicates overall shape of a rate-distortion curve for the given chunk.

8. The method of claim 1 further comprising, for the given chunk of the plural chunks:

with a scalable media encoder, encoding the given chunk as the plural scalable layers;

determining the curve information for the given chunk; and

storing the plural scalable layers of the given chunk and the curve information for the given chunk in storage for delivery to plural media clients.

9. The method of claim 1 wherein the network is the Internet, and wherein the curve information, the feedback from the media client, and the selected scalable layers are delivered over the Internet.

10. The method of claim 1 wherein the computing system than implements the media server is a distributed computing system that also includes a scalable media encoder.

11. A method of controlling delivery of media data by a media server across a network, wherein the media data is organized as plural chunks, a given chunk of the plural chunks being encoded as plural scalable layers for different quality levels with fine-grained scalability, the method comprising:

at a computing system that implements a media client, receiving curve information, wherein the curve information at least in part indicates distortion as a function of download time for the given chunk; and

with the computing system that implements the media client, determining a rate-distortion preference for the given chunk using at least some of the received curve information, wherein the determining the rate-distortion preference includes, for a segment of one or more chunks of the plural chunks, the segment including the given chunk;

determining a playback time constraint for the segment; and

for each of the one or more chunks in the segment, selecting a download time so as to reduce overall distortion among the one or more chunks in the segment while satisfying the playback time constraint for the segment; and

with the computing system that implements the media client, sending, for receipt by the media server across the network, the rate-distortion preference for the given chunk; and

receiving one or more scalable layers from among the plural scalable layers of the given chunk.

12. The method of claim 11 wherein the playback time constraint for the segment indicates a cumulative time duration, and wherein the selecting a download time for each of the one or more chunks in the segment comprises:

setting a weight parameter such that cumulative download time for the segment matches the cumulative time duration, wherein for each of the one or more chunks in the segment the weight parameter indicates the download time for the chunk according to the curve information for the chunk.

13. The method of claim 11 wherein the selecting a download time for each of the one or more chunks in the segment comprises:

splitting the segment into plural sub-segments; and

for each of the plural sub-segments, recursively;

determining a playback time constraint for the sub-segment; and

for each of one or more chunks in the sub-segment, selecting a download time so as to reduce overall distortion among the one or more chunks in the sub-segment while satisfying the playback time constraint for the sub-segment.

14. The method of claim 11 wherein for the given chunk of the plural chunks:

the plural scalable layers include a base layer and one or more residual layers;

the base layer is always received for the given chunk; and

zero or more of the one or more residual layers are received for the given chunk, depending on the rate-distortion preference.

15. The method of claim 11 wherein each of the plural chunks is encoded without dependencies on any other chunk of the plural chucks, and wherein the method further comprises, at the computing system that implements the media client, receiving a stream header for the media data organized as the plural chunks, the stream header including program description information for the media data organized as the plural chunks.

16. The method of claim 11 wherein the determining the rate-distortion preference for the given chunk includes determining a count of bits or range of bits for the given chunk, and wherein the sending the rate-distortion preference for the given chunk includes sending a request for the count of bits or range of bits for the given chunk.

17. The method of claim 11 wherein the method further comprises, for the given chunk of the plural chunks:

with a scalable media decoder, decoding the received one or more scalable layers for the given chunk; and

reconstructing a version of the given chunk from the decoded one or more scalable layers.

18. The method of claim 11 wherein the network is the Internet, and wherein the curve information, the rate-distortion preferences, and the selected scalable layers are delivered over the Internet.

19. The method of claim 11 wherein the computing system than implements the media client is a distributed computing system that also includes a scalable media decoder.

20. A computing system that implements a media client for controlling delivery of media data by a media server across a network, the system comprising:

a processor;

a controller for the media client, wherein the controller is adapted to;

receive a stream header for media data that includes plural chunks, each of the plural chunks being encoded as plural scalable layers for different quality levels with fine-grained scalability;

receive curve information for the plural chunks, wherein, for each of the plural chunks, the curve information for the chunk indicates distortion as a function of download time and a different rate-distortion point is associated with each of the plural scalable layers for the chunk;

determine rate-distortion preferences for the plural chunks, respectively, using the received curve information, wherein the controller is adapted to determine the rate-distortion preferences by, for a segment of one or more chunks of the plural chunks;

determining a playback time constraint for the segment; and

for each of the one or more chunks in the segment, selecting a download time so as to reduce overall distortion among the one or more chunks in the segment while satisfying the playback time constraint for the segment; and

send, for receipt by the media server across the network, the rate-distortion preferences for the plural chunks, respectively; and

for each of the plural chunks, receive one or more scalable layers from among the plural scalable layers for the chunk; and

a scalable media decoder adapted to decode the received scalable layers for the plural chunks.

21. The system of claim 20 wherein:

the controller is further adapted to determine the rate-distortion preferences by

for each of the one or more chunks in the segment, using the selected download time for the chunk to determine a count of bits or range of bits to request for the chunk.

22. The system of claim 21 wherein the playback time constraint for the segment indicates a cumulative time duration, and wherein the selecting a download time for each of the one or more chunks in the segment comprises:

setting a weight parameter such that cumulative download time for the segment matches the cumulative time duration, wherein for each of the one or more chunks in the segment the weight parameter indicates the download time for the chunk according to the curve information for the chunk.

23. The system of claim 20 wherein for each of the plural chunks, the plural scalable layers include a base layer and one or more residual layers.

24. A computing system that implements a media client to control delivery of media data by a media server across a network, wherein the media data is organized as plural chunks, each of the plural chunks being encoded as plural scalable layers for different quality levels with fine-grained scalability, the system comprising a processor and a controller adapted to perform a method comprising:

receiving, by the processor, curve information for the plural chunks, wherein, for each of the plural chunks, for the plural scalable layers of the chunk, the curve information at least in part indicates distortion as a function of download time; and

for each of the plural chunks;

determining a rate-distortion preference for the chunk using at least some of the received curve information, wherein the determining the rate-distortion preference includes, for a segment of one or more chunks of the plural chunks;

determining a playback time constraint for the segment, wherein the playback time constraint for the segment indicates a cumulative time duration; and

for each of the one or more chunks in the segment, selecting a download time so as to reduce overall distortion among the one or more chunks in the segment while satisfying the playback time constraint for the segment, wherein the selecting the download time for each of the one or more chunks in the segment comprises setting a weight parameter such that cumulative download time for the segment matches the cumulative time duration, and wherein for each of the one or more chunks in the segment the weight parameter indicates the download time for the chunk according to the curve information for the chunk;

sending the rate-distortion preference for the chunk; and

receiving one or more scalable layers from among the plural scalable layers of the chunk.

25. A computing system that implements a media client to control delivery of media data by a media server across a network, wherein the media data is organized as plural chunks, each of the plural chunks being encoded as plural scalable layers for different quality levels with fine-grained scalability, the system comprising a processor and a controller adapted to perform a method comprising:

receiving, by the processor, curve information for the plural chunks, wherein, for each of the plural chunks, for the plural scalable layers of the chunk, the curve information at least in part indicates distortion as a function of download time; and

for each of the plural chunks;

determining a rate-distortion preference for the chunk using at least some of the received curve information, wherein the determining the rate-distortion preference includes, for a segment of one or more chunks of the plural chunks;

determining a playback time constraint for the segment; and

for each of the one or more chunks in the segment, selecting a download time so as to reduce overall distortion among the one or more chunks in the segment while satisfying the playback time constraint for the segment, wherein the selecting the download time for each of the one or more chunks in the segment comprises splitting the segment into plural sub-segments and, for each of the plural sub-segments, recursively;

determining a playback time constraint for the sub-segment; and

for each of one or more chunks in the sub-segment, selecting a download time so as to reduce overall distortion among the one or more chunks in the sub-segment while satisfying the playback time constraint for the sub-segment;

sending the rate-distortion preference for the chunk; and

receiving one or more scalable layers from among the plural scalable layers of the chunk.