Device, system and method of real-time multimedia streaming

• US 5,768,527 A
• Filed: 04/23/1996
• Issued: 06/16/1998
• Est. Priority Date: 04/23/1996
• Status: Expired due to Term

• Current Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Original Assignee: Motorola, Inc. (Motorola Solutions, Inc.)
• Inventors: Eyuboglu, M. Vedat, Sridhar, Manickam R., Zhu, Qin-Fan
• Primary Examiner(s): Trammell, James P.
• Assistant Examiner(s): Nguyen, Cuong H.
• Application Number:
  US08/636,417
• Time in Patent Office: 784 Days
• Field of Search: 395/200.01, 395/200.02, 395/200.13, 395/200.61, 395/200.62, 395/200.65, 395/200.6, 370/231-232, 370/229-230, 370/241, 370/473, 371/32, 455/500, 455/507, 455/510, 455/18, 455/13.1, 455/70
• US Class Current: 709/231
• CPC Class Codes: H04L 1/0002 by adapting the transmission rate
• CPC Class Codes: H04L 1/1671 the supervisory signal being transmitted together with control information
• CPC Class Codes: H04L 1/1809 Selective-repeat protocols
• CPC Class Codes: H04L 1/1854 Scheduling and prioritising arrangements
• CPC Class Codes: H04L 1/1858 Transmission or retransmission of more than one copy of acknowledgement message
• CPC Class Codes: H04L 1/1887 Scheduling and prioritising arrangements
• CPC Class Codes: H04L 1/189 Transmission or retransmission of more than one copy of a message
• CPC Class Codes: H04L 12/6418 Hybrid transport
• CPC Class Codes: H04L 2012/6456 Channel and bandwidth allocation
• CPC Class Codes: H04L 2012/6467 Information loss recovery, e.g. error correction, prediction
• CPC Class Codes: H04L 2012/647 Frame Relay, X.25
• CPC Class Codes: H04L 2012/6472 Internet
• CPC Class Codes: H04L 2012/6475 N-ISDN, Public Switched Telephone Network [PSTN]
• CPC Class Codes: H04L 2012/6481 Speech, voice
• CPC Class Codes: H04L 2012/6483 Video, e.g. MPEG
• CPC Class Codes: H04L 2012/6497 Feedback to the source
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04L 67/61 taking into account QoS or priority requirements
• CPC Class Codes: H04L 69/329 in the application layer [OSI layer 7]
• CPC Class Codes: H04L 9/40 : Network security protocols
• CPC Class Codes: H04N 19/89 : involving methods or arrangements for detection of transmission errors at the decoder
• CPC Class Codes: H04N 19/895 : in combination with error concealment
• CPC Class Codes: H04N 21/21 : Server components or server architectures
• CPC Class Codes: H04N 21/23 : Processing of content or additional data; Elementary server operations; Server middleware
• CPC Class Codes: H04N 21/44209 : Monitoring of downstream path of the transmission network originating from a server, e.g. bandwidth variations of a wireless network arrangements for maintenance or administration in data switching networks involving bandwidth and capacity management H04L41/0896
• CPC Class Codes: H04N 21/6375 : for requesting retransmission , e.g. of data packets lost or corrupted during transmission from serverARQ protocols H04L1/18; implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP] H04L69/16
• CPC Class Codes: H04N 21/6377 : directed to server one-way streaming services wherein the source is controlled by the destination H04L65/613
• CPC Class Codes: H04N 21/643 : Communication protocols network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP], H04L65/65
• CPC Class Codes: H04N 21/658 : Transmission by the client directed to the server
• CPC Class Codes: H04N 7/17336 : Handling of requests in head-ends
• CPC Class Codes: H04N 7/24 : Systems for the transmission of television signals using pulse code modulation H04N21/00 takes precedence
• CPC Class Codes: Y02D 30/50 : in wire-line communication networks, e.g. low power modes or reduced link rate

Abstract

The present invention provides a device (200, 300), system (400, 500) and method for real-time streaming of a multimedia file stored in a remote server over a packet network to a multimedia client connected to the same network via a low-speed access link such as an analog telephone line. The invention operates to provide significantly improved QoS with respect to both quality and delay.

Claims

1. A client device for receiving and in real-time playing out a multimedia file stored in a remote server, said client device and server being connected to a packet network, said client device comprising:

A) a packet buffer operably coupled to store incoming packets sent by the remote server,

B) a packet processor operably coupled to the packet buffer for detecting lost packets, and

C) a quality of service, QoS, manager operably coupled to the packet processor for sending retransmission requests to the remote server, wherein said retransmission requests are sent upon detection of a lost packet and request the retransmission of multiple copies of the lost packet, and wherein a number of multiple copies requested depends on an importance of the lost packet.

2. The client device of claim 1, wherein the number of copies requested further depends on a remaining number of retransmission attempts allowed for the lost packet.

3. The client device of claim 2, wherein the remaining number of retransmission attempts allowed for the lost packet is obtained by dividing a difference between a presentation time of the packet and a current time by a roundtrip delay between the client device and the remote server.

4. The client device of claim 1, wherein multiple copies of the retransmission request are sent by the client to the remote server.

5. The client device of claim 1, wherein the multiple number of copies requested further depends on a bandwidth budget and an end-to-end information loss rate.

6. The client device of claim 1 wherein the QoS manager further sends rate control messages to the remote server to control the bitrate of the information stream, wherein said rate control messages are sent using a predetermined scheme based on said retransmission requests, the bandwidth budget and the end-to-end information loss rate.

7. The client device of claim 6, wherein the said number of copies for retransmission and the rate control messages are sent based on the following relationship:

space="preserve" listing-type="equation">R'"'"'+Σ

.sub.1≦

i≦

M (Σ

.sub.1≦

j≦

Li q.sup.j c.sub.ij)R'"'"'.sub.i ≦

G,

where

M is a number of the substreams the original multimedia file contains;

L_i is a number of retransmission attempts for substream i;

q is the end-to-end information loss rate;

c_ij is the number of copies for retransmission for substream i in the jth retransmission attempt;

R'"'"'_i is the streaming bitrate of substream i;

R'"'"'=Σ

₁≦

i≦

M R'"'"'_i ;

G is the bandwidth budget at the server.

8. A client device for receiving and in real-time playing out a multimedia file stored in a remote server attached to a packet network, said client device being connected to a packet network via a low-speed access link and an access gateway, said client device comprising:

A) a packet buffer operably coupled to store incoming packets sent by the remote server,

B) a packet processor operably coupled to the packet buffer for detecting a lost packet, and

C) a quality of service, QoS, manager operably coupled to the packet processor for sending retransmission requests to the remote server, wherein said retransmission requests are sent based on an available bitrate on the low-speed access link and an information loss rate on the packet network.

9. The client device of claim 8, wherein multiple copies of the lost packet are requested in said retransmission requests, and the number of copies requested depends on an importance of the lost packet.

10. The client device of claim 9, wherein a number of multiple copies requested further depends on a remaining number of retransmission attempts allowed for the lost packet.

11. The client device of claim 10, wherein the remaining number of retransmission attempts allowed for the lost packet is obtained by dividing a difference between a presentation time of a packet and a current time by a roundtrip delay between the client device and the remote server.

12. The client device of claim 9 wherein multiple copies of the retransmission requests are sent by the client to the remote server.

13. The client device of claim 8 wherein the QoS manager further sends rate control messages to the remote server to control the bitrate of the information stream, wherein said rate control messages are sent using a predetermined scheme based on an available bitrate on the low-speed access link and an information loss rate on the packet network.

14. The client device of claim 13, wherein a number of copies for retransmission and the rate control messages are sent based on an equation of a form:

space="preserve" listing-type="equation">R'"'"'+Σ

.sub.1≦

i≦

M (Σ

.sub.1≦

j≦

Li q.sup.j c.sub.ij)R'"'"'.sub.i ≦

min(G,B/(1-p)),

where

M is a number of substreams the multimedia file contains;

L_i is a number of retransmission attempts for substream i;

q is an end-to-end information loss rate;

c_ij is a number of copies for retransmission for substream i in a jth retransmission attempt;

R'"'"'_i is a streaming bitrate of substream i;

R'"'"'=Σ

₁≦

i≦

M R'"'"'_i ;

G is a bandwidth budget at the server;

B is a bitrate on the low-speed access link;

p is an information loss rate on the packet network.

15. The client device of claim 8, wherein the information loss rate on the packet network is determined from an end-to-end information loss rate and an information loss rate on the low-speed access link.

16. The client device of claim 15, wherein said information loss rate on the low-speed access link is obtained using an error detection code.

17. The client device of claim 8, wherein the available bitrate on the low-speed access link is provided by a transmission device connected to the low-speed access link.

18. The client device of claim 17 wherein the said transmission device is a voiceband modem.

19. The client device of claim 17 wherein the transmission device has an auxiliary control channel in addition to a data channel through which control information is exchanged between the client device and the transmission device.

20. The client device of claim 19 wherein the auxiliary channel is used to communicate an estimate of a bit error rate of a low-speed point-to-point access link to the client device.

21. A system for streaming real-time multimedia information from a server to a client device over a packet network, consisting of:

A) a server for providing a bitstream by streaming the real-time multimedia information and performing retransmission, said server comprising;

A1) a packetizer for converting the bitstream representing the real-time multimedia information into transmission packets;

A2) a packet buffer for storing said transmission packets;

A3) a packet transmitter for transmitting said transmission packets;

A4) a control message processor for receiving control messages sent by the client device and for initiating retransmission of multiple copies of a packet upon receipt of a retransmission request from the client device;

B) a packet network operably connected to the server, for carrying the bitstream to a client device;

C) a client device having a multimedia player for playing out the bitstream in realtime, said client device comprising;

C1) a packet buffer operably coupled to store incoming packets of the multimedia stream;

C2) a packet processor, operably coupled to the packet buffer, for detecting lost packets, and

C3) a quality of service, QoS, manager operably coupled to the packet processor for sending retransmission requests to the remote server, wherein said retransmission requests are sent upon detection of a lost packet and request retransmission of multiple copies of the lost packet, wherein a number of copies requested depends on an importance of the lost packet.

22. The system of claim 21 wherein the number of copies requested by a client further depends on a remaining number of retransmission attempts allowed for the lost packet.

23. The system of claim 21 wherein multiple copies of the retransmission request are sent by the client device to the remote server.

24. The system of claim 21 wherein the number of copies requested by the client device further depends on a bandwidth budget and an end-to-end information loss rate.

25. The system of claim 21 wherein the QoS manager in the client device further sends rate control messages to the server to control a bitrate of the information stream, wherein said rate control messages are sent using a predetermined scheme based on said retransmission requests, a predetermined bandwidth budget and the end-to-end information loss rate.

26. A system for streaming real-time multimedia information stored in a remote server attached to a packet network, to a client device connected to the packet network via a low-speed access link and an access gateway, consisting of:

A) a server for providing a bitstream by streaming the real-time multimedia information and performing retransmission, said server comprising;

A1) a packetizer for converting the bitstream representing the real-time multimedia information into transmission packets;

A2) a packet buffer for storing said transmission packets;

A3) a packet transmitter for transmitting said transmission packets;

A4) a control message processor for receiving control messages sent by the client device and for initiating the retransmission of a packet upon receipt of a retransmission request from the client device;

B) a packet network operably connected to the server, for carrying the bitstream to a client device;

C) a client device having a multimedia player for playing out the bitstream in realtime, said client device comprising;

C1) a packet buffer operably coupled to store incoming packets of the bitstream;

C2) a packet processor, operably coupled to the packet buffer, for detecting lost packets, and

C3) a quality of service, QoS, manager operably coupled to the packet processor for sending retransmission requests to the remote server, wherein said retransmission requests are sent based on an available bitrate on the low-speed access link and an information loss rate on the packet network.

27. The system of claim 26, wherein multiple copies of a lost packet are requested in said retransmission request sent by the client device, and a number of copies requested depends on an importance of the lost packet, and the server in response initiates retransmission of multiple copies of a requested packet.

28. The system of claim 27 wherein the number of copies requested by the client device further depends on a remaining number of retransmission attempts allowed for the lost packet.

29. The system of claim 27 wherein multiple copies of the retransmission request are sent by the client device to the remote server.

30. The system of claim 26 wherein the QoS manager in the client device further sends rate control messages to the remote server to control a bitrate of the bitstream, wherein said rate control messages are sent using a predetermined scheme based on an available bitrate on the low-speed access link and an information loss rate on the packet network.

31. The system of claim 30, wherein the said information loss rate on the packet network is determined from an end-to-end information loss rate and an information loss rate on the low-speed access link.

32. The system of claim 31, wherein said information loss rate on the low-speed access link is obtained using an error detection code on a point-to-point access link.

33. The system of claim 26, wherein said available bitrate on the low-speed access link is provided by a transmission device connected to a low-speed access link.

34. The system of claim 33 wherein the said transmission device is a voiceband modem.

35. A system for streaming real-time multimedia information from a server to a client device over a packet network, consisting of:

A) a server for providing a bitstream by streaming the real-time multimedia information and performing retransmission, said server comprising;

A1) a packetizer for converting the bitstream representing the real-time multimedia information into transmission packets;

A2) a packet buffer for storing said transmission packets;

A3) a packet transmitter for transmitting said transmission packets;

A4) a quality of service, QoS, optimizer for receiving control messages sent by the client device and for initiating the retransmission of multiple copies of a packet upon receipt of a retransmission request from the client device and for determining how many copies of a lost packet to transmit based on an importance of the lost packet;

B) a packet network operably connected to the server, for carrying the bitstream to a client device;

C) a client device having a multimedia player for playing out the bitstream in real-time, said client device comprising;

C1) a packet buffer operably coupled to store incoming packets of the bitstream;

C2) a packet processor, operably coupled to the packet buffer, for detecting lost packets, and

C3) a QoS manager operably coupled to the packet processor for sending retransmission requests to the remote server, wherein said retransmission requests are sent upon detection of a lost packet.

36. The system of claim 35 wherein a number of copies of the retransmitted packet sent by the server further depends on a remaining number of retransmission attempts allowed for the lost packet.

37. The system of claim 35 wherein multiple copies of the retransmission request are sent by the client device to the server.

38. The system of claim 35 wherein a number of copies sent by the server further depends on a bandwidth budget and an end-to-end information loss rate provided by the client device.

39. The system of claim 35 wherein the QoS optimizer in the server further controls the bitrate of the information stream using a predetermined scheme based on said retransmission requests, the bandwidth budget and the end-to-end information loss rate provided by the client.

40. A system for streaming real-time multimedia information stored in a remote server attached to a packet network, to a client device connected to the packet network via a low-speed access link and an access gateway, consisting of:

A) the remote server for providing a bitstream by streaming the real-time multimedia information and performing retransmission, said remote server comprising;

A1) a packetizer for converting the bitstream representing the real-time multimedia information into transmission packets;

A2) a packet buffer for storing said transmission packets;

A3) a packet transmitter for transmitting said transmission packets; and

A4) a QoS optimizer for receiving control messages sent by the client device and for initiating the retransmission of a packet upon receipt of a retransmission request from the client device based on information on an available bitrate on the low-speed access link and an information loss rate on the packet network received from the client device;

B) a packet network operably connected to the remote server, for carrying the bitstream to a client device;

C) a client device having a multimedia player for playing out the bitstream in realtime, said client device comprising;

C1) a packet buffer operably coupled to store incoming packets of the bitstream;

C2) a packet processor, operably coupled to the packet buffer, for detecting lost packets, and

C3) a QoS manager operably coupled to the packet processor for sending retransmission requests and information on an available bitrate on the low-speed access link and an information loss rate on the packet network to the remote server.

41. The system of claim 40, wherein multiple copies of a lost packet requested by the client device is sent by the remote server, and a number of copies sent depends on an importance of the lost packet.

42. The system of claim 41 wherein the number of copies sent by the remote server further depends on a remaining number of retransmission attempts allowed for the lost packet.

43. The system of claim 41 wherein multiple copies of a retransmission request are sent by the client device to the remote server.

44. The system of claim 40 wherein the QoS optimizer in the server further controls a bitrate of the bitstream using a predetermined scheme based on an available bitrate on the low-speed access link and an information loss rate on the packet network provided by the client device.

45. The system of claim 40, wherein the said information loss rate on the packet network is determined by the client device from an end-to-end information loss rate and an information loss rate on the low-speed access link.

46. The system of claim 45, wherein said information loss rate on the low-speed access link is obtained using an error detection code.

47. The system of claim 40, wherein the said available bitrate on the low-speed access link is provided by a transmission device connected to the low-speed access link.

48. The system of claim 47 wherein the said transmission device is a voiceband modem.

49. A method for receiving and in real-time playing out, by a client device, a multimedia file stored in a remote server, wherein the client device and server are connected to a packet network, and the client device includes a packet buffer, a packet processor, and a quality of service QoS manager, comprising the steps of:

A) storing incoming packets sent by the remote server in the packet buffer,

B) detecting lost packets using the packet processor, and

C) sending, utilizing the QoS manager, retransmission requests to the remote server, wherein said retransmission requests are sent upon detection of a lost packet and requesting the retransmission of multiple copies of the lost packet, and wherein a number of multiple copies requested is in accordance with an importance of the lost packet.

50. The method of claim 49, wherein the number of multiple copies requested further depends on a remaining number of retransmission attempts allowed for the lost packet.

51. The method of claim 50, wherein the remaining number of retransmission attempts allowed for the lost packet is obtained by dividing a difference between a presentation time of the packet and a current time by a roundtrip delay between the client device and the remote server.

52. The method of claim 49, wherein multiple copies of the retransmission request are sent by the client device to the remote server.

53. The method of claim 49, wherein the multiple number of copies requested further depends on a bandwidth budget and an end-to-end information loss rate.

54. The method of claim 49, wherein the QoS manager further sends rate control messages to the remote server to control the bitrate of the bitstream, wherein said rate control messages are sent using a predetermined scheme based on said retransmission requests, the bandwidth budget and an end-to-end information loss rate.

55. The method of claim 54, wherein said number of multiple copies for retransmission and the rate control messages are sent based on an equation of a form:

space="preserve" listing-type="equation">R'"'"'+Σ

.sub.1≦

i≦

M (Σ

.sub.1≦

j≦

Li q.sup.j c.sub.ij)R'"'"'.sub.i ≦

G,

where

M is a number of the substreams the original multimedia file contains;

L_i is a number of retransmission attempts for substream i;

q is an end-to-end information loss rate;

c_ij is a number of copies for retransmission for substream i in a jth retransmission attempt;

R'"'"'_i is a streaming bitrate of substream i;

R'"'"'=Σ

₁≦

i≦

M R'"'"'_i ;

G is a bandwidth budget at the server.

56. A server for transmitting a multimedia file stored in the server to a client device, said client device and the server being connected to a packet network, said server comprising:

A) a rate scaler, operably coupled to receive the multimedia file and for providing an output of input multimedia substreams with requested data bitrates;

B) a packetizing unit, operably coupled to the rate scaler, for packetizing and storing the output of the rate scaler using a predetermined scheme,

C) a packet transmitter, operably coupled to the packetizing unit and a feedback unit, for, where the feedback unit lacks an output, fetching packets from the packetizing unit in a sequence generated and transmitting the packets to the packet network at a streaming rate substantially equal to a requested data bitrate, and where a retransmission request is initiated by the feedback unit, fetching a packet specified by a retransmission request and sending a requested number of copies of the packet; and

D) the feedback unit, operably coupled to send the multimedia file to the packet transmitter, receive and decode feedback messages and one of;

signaling the packet transmitter to retransmit a requested number of copies of a packet and forwarding a received message to the rate scaler.

57. The server of claim 56 wherein the feedback unit is a feedback message processor.

58. A server for transmitting a multimedia file stored in the server to a client device, said client device and the server being connected to a packet network, said server comprising:

A) a rate scaler, operably coupled to receive the multimedia file and for providing an output of input multimedia substreams with requested data bitrates;

B) a packetizing unit, operably coupled to the rate scaler, for packetizing and storing the output of the rate scaler using a predetermined scheme,

C) the packet transmitter, operably coupled to the packetizing unit and a QoS optimizer, for transmitting packets along the packet network;

D) a feedback unit, operably coupled to the packet network, the rate scaler, and to receive a predetermined bandwidth budget G, for at least receiving and decoding feedback messages and one of;

signaling the packet transmitter to retransmit a requested number of copies of a packet and forwarding a received message to the rate scaler.

59. The server of claim 58 wherein the feedback unit includes:

A) a QoS optimizer, operably coupled to the rate scaler, a feedback message processor, and to receive a predetermined bandwidth budget G, for, where data for a multimedia file consists of M substreams, M an integer greater than or equal to two, R_i is an original data rate of substream i, and c_ij ≧

0 represents a number of copies of a lost packet requested for retransmission for substream i in a jth retransmission attempt, optimizing transmittal of packets in accordance with an optimizing scheme that provides a trade-off between streaming rates R'"'"'_i of substreams and the number of copies requested for retransmission according to an equation of a form;

space="preserve" listing-type="equation">R'"'"'+Σ

.sub.1≦

i≦

M (Σ

.sub.1≦

j≦

Li p.sup.j c.sub.ij)R'"'"'.sub.i ≦

G,

where

space="preserve" listing-type="equation">R'"'"'.sub.i ≦

R.sub.i

and

space="preserve" listing-type="equation">R'"'"'=Σ

.sub.1≦

i≦

M R'"'"'.sub.i,

and p is an information loss rate in the packet network, and L_i is a number of retransmission attempts for substream i; and

B) a feedback message processor, operably coupled to send the multimedia file to the packet transmitter for transmission to the client device via the packet network, receive feedback messages sent by the client device via the packet network and decode the feedback messages, wherein,

B1) where the feedback message is a retransmission request message, the feedback message processor passes the identity of the requested packet and the number of copies for retransmission to the packet transmitter, which fetches the requested packet from the packet buffer and transmits the requested number of copies of the packet to the packet network;

B2) where the received message is a rate control message, the feedback message processor forwards the received message to the rate scaler.