System and method for delivery of dynamically scalable audio/video content over a network

• US 6,789,123 B2
• Filed: 12/28/2001
• Issued: 09/07/2004
• Est. Priority Date: 12/28/2001
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Li, Jin, Zhou, Jianping
• Primary Examiner(s): Cuchlinski, Jr., William A.
• Assistant Examiner(s): Phan, Tam T.
• Application Number:
  US09/683,427
• Publication Number:
  US 20030135631A1
• Time in Patent Office: 984 Days
• Field of Search: 709/231, 709/232, 709/207, 709/233, 709/247, 725/87, 725/90, 725/98, 725/145, 370/230, 370/231, 370/232, 370/233, 370/234, 370/235, 370/238, 370/465, 370/468
• US Class Current: 709/231
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/612 for unicast
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 65/752 adapting media to network capabilities
• CPC Class Codes: H04L 65/80 Responding to QoS

Abstract

A system and process for streaming delivery of dynamically scalable media content over a network, such as, for example, the Internet or a wireless network, while automatically accounting for both fluctuating network bandwidth and packet loss. A system of rate-distortion based packet selection and organization is used to maximize the quality of streamed media files that have been encoded using any conventional scalable encoder. Media file quality is maximized for available bandwidth by scoring packets comprising encoded media files based on their contribution to the quality of a reconstructed media file, then preferentially transmitting those packets having the highest scores. In addition, where packets are lost during transmission, those packets that will provide the maximum rate-distortion decrease, are preferentially retransmitted prior to other lost packets which, if transmitted in the same time slot, would provide a lesser rate-distortion decrease.

Claims

1. A system for streaming automatically and dynamically scalable media content over a network, comprising:

analyzing a scalably encoded media file comprising at least one layered bitstream;

determining a rate-distortion decrease attributable to each of at least one data units comprising each bitstream;

computing a rate-distortion-based score for each data unit for a current network transmission timeslot;

dynamically organizing at least one highest score data unit into a least one network packet in order of decreasing score, from highest to lowest; and

transmitting the at least one network packet from a server to at least one client based on available network bandwidth.

2. The system of claim 1 further comprising computing a new rate-distortion-based score for each data unit for each new network transmission timeslot.

3. The system of claim 2 further comprising:

dynamically organizing at least one highest score data unit into a least one network packet in order of decreasing score, from highest to lowest for each new network transmission timeslot; and

transmitting the at least one network packet from a server to at least one client based on available network bandwidth for each new network transmission timeslot.

4. The system of claim 1 wherein the rate-distortion-based score computed for each data unit proportionally decreases as a size of each data unit increases.

5. The system of claim 1 further comprising automatically determining a network packet loss ratio.

6. The system of claim 1 wherein the rate-distortion-based score computed for each data automatically accounts for the network packet loss ratio.

7. The system of claim 1 further comprising receiving the at least one network packet at the at least one client.

8. The system of claim 7 wherein the at least one client automatically decodes the at least one network packet.

9. The system of claim 8 wherein each decoded network packet is streamed to a media player application.

10. The system of claim 8 wherein each decoded network packet is streamed to a media playback buffer which in turn provides the decoded network packets to a media player application after a predetermined delay.

11. The system of claim 1 further comprising automatically including forward error correction information in at least one network packet.

12. The system of claim 11 further comprising automatically correcting for at least one lost packet using the forward error correction information included in at least one network packet.

13. A computer-implemented process for providing network aware rate-distortion optimization for streaming media files over a network, comprising:

encoding at least one media file using a scalable encoder, wherein each encoded media file further comprises at least one layered bitstream, each bitstream comprised of at least one data unit;

scoring the data units comprising each layered bitstream with respect to a calculated rate-distortion decrease for each data unit;

dynamically arranging data units within network packets in order of highest score to lowest score;

transmitting at least one of the network packets from at least one server to at least one client, beginning with a network packet having the highest score data units; and

continuing to transmit all available network packets in decreasing order of contained highest score data units until a measured network bandwidth has been fully utilized.

14. The computer-implemented process of claim 13 wherein scoring the data units further comprises decreasing the score for each data unit in direct proportion to a byte size of each data unit to reflect a bandwidth cost of transmitting each data unit.

15. The computer-implemented process of claim 13 wherein scoring the data units further comprises modifying the score for each data unit to reflect a contribution of a reliance factor for each data unit, and wherein the reliance factor accounts for any relationships between data units wherein one or more current data units require receipt of one or more prior data units by a client before the current data units may be decoded.

16. The computer-implemented process of claim 13 wherein scoring the data units further comprises modifying the score for each data unit to reflect a contribution of a sent status for each data unit, and wherein the sent status indicates whether a particular data unit has already been transmitted within a network packet from the at least one server.

17. The computer-implemented process of claim 16 wherein the sent status further includes an indication of whether a NACK has been received by the at least one server following the attempted transmission of at least one network packet.

18. The computer-implemented process of claim 17 wherein the score for a particular data unit is set to zero if a network packet containing that data unit has been transmitted and no NACK has been received, and wherein data units having a score of zero are not retransmitted.

19. The computer-implemented process of claim 13 wherein scoring the data units further comprises modifying the score for each data unit to reflect a probability of on-time delivery for a data unit within a transmitted network packet, and wherein the score for each data unit decreases in direct proportion to a probability that that data unit can be delivered to a client in time for uninterrupted playback of the media file.

20. The computer-implemented process of claim 13 wherein scoring the data units further comprises modifying the score for each data unit to reflect a contribution of a balance factor for indicating an importance of near future time slots, and wherein those data units which must be delivered sooner to provide for uninterrupted playback of the media file are deemed more important than those data units which are not needed until a later time slot for uninterrupted playback of the media file.

21. A computer-readable medium having computer executable instructions for automatically and dynamically streaming scalable media content over a network, said computer executable instructions comprising:

scalably encoding a media file;

determining a rate-distortion for data units comprising the scalably encoded media file;

computing a score for each data unit based on adistortion decrease for each data unit and a size of each data unit, and wherein the score is recomputed for each timeslot of a network transmission;

arranging at least one of the data units into at least one network packet, wherein data units are placed into packets based on the score for each data unit, with higher scoring data units being given priority over lower scoring data units for each timeslot of the network transmission;

determining available network bandwidth for each timeslot of a network transmission between a server and at least one client; and

transmitting all available packets from at least one server to at least one client beginning with a packet including the highest score data units, and continuing with packets having progressively lower data units until the available network bandwidth has been fully utilized for each timeslot of a network transmission.

22. The computer-readable medium of claim 21 wherein the score computed for each data unit further comprises any of a proportional decrease in score as a size of each data unit increases;

a proportional increase in score as an automatically computed network packet loss ratio increases;

a proportional increase in score to reflect a contribution of a reliance factor for each data unit, and wherein the reliance factor accounts for any relationships between data units wherein one or more current data units require receipt of one or more prior data units by a client before the current data units may be decoded;

setting the score to zero for a particular data unit where a sent status indicates that the particular data unit has already been transmitted within a network packet;

a proportional increase in score to reflect receipt of a NACK following the attempted transmission of at least one network packet;

a proportional increase in score to reflect a probability of on-time delivery for a data unit within a transmitted network packet, and wherein the score for each data unit decreases in direct proportion to a probability that that data unit can be delivered to a client in time for uninterrupted playback of the media file; and

a proportional increase in score to reflect a contribution of a balance factor for indicating an importance of near future time slots, and wherein those data units which must be delivered sooner to provide for uninterrupted playback of the media file are deemed more important than those data units which are not needed until a later time slot for uninterrupted playback of the media file.

23. The computer-readable medium of claim 21 wherein the at least one client automatically decodes the at least one network packet.

24. The computer-readable medium of claim 23 wherein each decoded network packet is streamed to a media player application.

25. The computer-readable medium of claim 23 wherein each decoded network packet is streamed to a media playback buffer which in turn provides the decoded network packets to a media player application after a predetermined delay.

26. The computer readable medium of claim 21 further comprising automatically including forward error correction information in at least one network packet.

27. The computer-readable medium of claim 26 further comprising automatically correcting for at least one lost packet using the forward error correction information included in at least one network packet.