Selective retransmission for efficient and reliable streaming of multimedia packets in a computer network

• US 5,918,002 A
• Filed: 03/14/1997
• Issued: 06/29/1999
• Est. Priority Date: 03/14/1997
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Cannon, Anthony William, Ravi, Hemanth Srinivas, Vellanki, Srinivas Prasad, Klemets, Anders Edgar
• Primary Examiner(s): HUA, LY
• Application Number:
  US08/818,644
• Time in Patent Office: 837 Days
• Field of Search: 371/32, 371/33, 371/35, 395/192.16, 455/7
• US Class Current: 714/18
• CPC Class Codes: H04L 1/1838 for semi-reliable protocols, e.g. for less sensitive applications such as streaming video buffer level management for video bitstream receiver H04N21/44004
• CPC Class Codes: H04L 1/1877 for semi-reliable protocols, e.g. for less sensitive applications like streaming video buffer level management for video bitstream control arrangements H04N21/44004
• CPC Class Codes: H04L 2012/6459 Multiplexing, e.g. TDMA, CDMA
• CPC Class Codes: H04L 2012/6489 Buffer Management, Threshold setting, Scheduling, Shaping
• CPC Class Codes: H04L 2012/6494 Silence suppression
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/1108 Web based protocols, e.g. webRTC
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/568 Storing data temporarily at an intermediate stage, e.g. caching
• CPC Class Codes: H04L 69/28 Timers or timing mechanisms used in protocols
• CPC Class Codes: H04L 69/329 in the application layer [OSI layer 7]
• CPC Class Codes: H04L 9/40 Network security protocols

Abstract

An efficient and reliable transmission protocol for transmitting multimedia streams from a server to a client computer over a diverse computer network including local area networks (LANs) and wide area networks (WANs) such as the internet. The client computer includes a playout buffer for temporary storage of incoming data packets. When the client computer detects that a data packet has not arrived at said client computer by an expected time of arrival (ETA), a round trip time for the data packet is computed. The round trip time is an estimate of a period beginning from the time a retransmission request is sent to from the client computer to the stream server till the time a copy of the missing data packet is received at the client computer from the stream server in response to the retransmission request. If the round trip time is less than the time remaining before the missing packet is no longer useful to the on-demand application, then a retransmission request packet is sent to the server. Conversely if the round trip time is greater than the time remaining, i.e., the missing packet is likely to arrive after the usefulness of the packet has expired, then sending a retransmission request is likely to result in the late arrival of the missing data packet. Accordingly, the missing packet is discarded. This selective retransmission protocol can also be practiced with dynamic bandwidth selection wherein the transmission rate is dynamically matched to the available bandwidth capacity of the network connection between the server and the client computer.

Claims

1. In a client computer, a method for selectively retransmitting missing data packets of a data stream from a server to said client computer via an unreliable computer network, the method useful in association with an on-demand application of said client computer, the client computer including a playout buffer for storing a plurality of data packets of said data stream being transmitted from the server to the client computer, the method comprising:

(a) detecting that a first said missing data packet has not arrived at said client computer by a first expected time of arrival (ETA);

(b) computing a first Round_-- Trip_-- Time for said first missing data packet, the first Round_-- Trip_-- Time being an estimate of a period beginning from a time a first retransmission request is sent from the client computer to the stream server till a time a copy of said first missing data packet is received at the client computer from the stream server in response to the first retransmission request;

(c) computing a first time remaining before said first missing data packet is due for processing by said on-demand application; and

(d) if said first Round_-- Trip_-- Time is less than said first time, then sending said first retransmission request packet to said server for said first missing data packet.

2. The method of claim 1 wherein if said first missing data packet is an important data packet, then the first retransmission request packet is sent to said server for said important first missing data packet, even if said first Round_-- Trip_-- Time is greater than the first time remaining before said first missing data packet is due for processing by said on-demand application.

3. The method of claim 2 wherein said data stream is a video stream and said first missing data packet includes an I video frame.

4. The method of claim 2 wherein said data stream is a video stream and said first missing data packet includes a base layer video frame.

5. The method of claim 2 wherein said data stream is an audio stream, said first missing data packet is an audio packet and a data size of said audio packet is greater than a first difference between the first Round_-- Trip_-- Time and the first time remaining before said first missing data packet is due for processing by said on-demand application.

6. The method of claim 1 further comprising:

(e) detecting that a second said missing data packet has not arrived at said client computer by a second ETA; and

(f) computing a second Round_-- Trip_-- Time for said second missing data packet, the second Round_-- Trip_-- Time being an estimate of a period beginning from a time a second retransmission request is sent from the client computer to the stream server till a time a copy of said second missing data packet is received at the client computer from the stream server in response to the second retransmission request;

(g) computing a second time remaining before said second missing data packet is due for processing by said on-demand application; and

(h) if said second Round_-- Trip_-- Time is less than the second time, and

if said second missing data packet has a higher priority than said first missing data packet, then sending the second retransmission request packet to said server for said second missing data packet prior to step (d);

else if second missing data packet has a lower priority than said first missing data packet, then sending the second retransmission request packet to said server for said second missing data packet after step (d).

7. The method of claim 6 wherein said priority of first and second missing data packets is based on the respective data type of said missing data packets.

8. The method of claim 6 wherein said first missing data packet includes an I video frame, said second missing data packet includes a P video frame, and hence said first missing data packet has a higher priority than said second missing data packet.

9. The method of claim 6 wherein said first missing data packet includes a base layer video frame, said second missing data packet includes an enhancement layer video frame, and hence said first missing data packet has a higher priority than said second missing data packet.

10. The method of claim 6 wherein a first difference between the first Round_-- Trip_-- Time and the first time remaining is less than a second difference between the second Round_-- Trip_-- Time and the second time remaining, and hence said first missing data packet has a higher priority than said second missing data packet.

11. The method of claim 10 wherein said data stream is a video data stream and wherein said first and said second missing data packets include video frames.

12. The method of claim 10 wherein said data stream is an annotation stream and wherein said first and second missing data packets include annotation frames.

13. A client computer useful in association with a stream server, said client computer coupled to said stream server via an unreliable computer network, the client computer comprising:

a playout buffer for storing a plurality of data packets of said data stream being transmitted from the server to the client computer, and wherein if a first said missing data packet has not arrived at said client computer by a first expected time of arrival (ETA) and if a first Round_-- Trip_-- Time of said first missing data packet is less than a first time remaining before said first missing data packet is due for processing by an on-demand application of said client computer, the first Round_-- Trip_-- Time being an estimate of a period beginning from a time a first retransmission request is sent to from the client computer to the stream server till a time a copy of said first missing data packet is received at the client computer from the stream server in response to the first retransmission request, then said first retransmission request packet is sent to said server for said first missing data packet.

14. The client computer of claim 13 wherein if said first missing data packet is an important data packet, then the first retransmission request packet is sent to said server for said important first missing data packet, even if said first Round_-- Trip_-- Time is greater than the first time remaining before said first missing data packet is due for processing by said on-demand application.

15. The client computer of claim 14 wherein said data stream is a video stream and said first missing data packet includes an I video frame.

16. The client computer of claim 14 wherein said data stream is a video stream and said first missing data packet includes a base layer video frame.

17. The client computer of claim 14 wherein said data stream is an audio stream, said first missing data packet is an audio packet and a data size of said audio packet is greater than a first difference between the first Round_-- Trip_-- Time and the first time remaining before said first missing data packet is due for processing by said on-demand application.

18. The client computer of claim 13 wherein if a second said missing data packet has not arrived at said client computer by a second ETA and if a second Round_-- Trip_-- Time is less than a second time remaining before said second missing data packet is due for processing by said on-demand application, the second Round_-- Trip_-- Time being an estimate of a period beginning from a time a second retransmission request is sent to from the client computer to the stream server till a time a copy of said second missing data packet is received at the client computer from the stream server in response to the second retransmission request, then:

if said second missing data packet has a higher priority than said first missing data packet, then sending the second retransmission request packet to said server prior to sending the first retransmission request packet;

else if second missing data packet has a lower priority than said first missing data packet, then sending the second retransmission request packet to said server after to sending the first retransmission request packet.

19. The client computer of claim 18 wherein said priority of first and second missing data packets is based on the respective data type of said missing data packets.

20. The client computer of claim 18 wherein said first missing data packet includes an I video frame, said second missing data packet includes a P video frame, and hence said first missing data packet has a higher priority than said second missing data packet.

21. The client computer of claim 18 wherein said first missing data packet includes a base layer video frame, said second missing data packet includes an enhancement layer video frame, and hence said first missing data packet has a higher priority than said second missing data packet.

22. The client computer of claim 18 wherein a first difference between the first Round_-- Trip_-- Time and the first time remaining is less than a second difference between the second Round_-- Trip_-- Time and the second time remaining, and hence said first missing data packet has a higher priority than said second missing data packet.

23. The client computer of claim 22 wherein said data stream is a video data stream and wherein said first and said second missing data packets include video frames.

24. The client computer of claim 22 wherein said data stream is an annotation stream and wherein said first and second missing data packets include annotation frames.

25. A method comprising:

detecting that a first missing data packet has not arrived at a client computer by a first expected time of arrival (ETA);

computing a first Round_-- Trip_-- Time for the first missing data packet, the first Round_-- Trip_-- Time being an estimate of a period beginning from a time a first retransmission request is sent from the client computer to a server computer till a time a copy of the first missing data packet is received at the client computer from the server computer in response to the first retransmission request;

computing a first time remaining before the first missing data packet is due for processing by an application on the client computer;

sending a first retransmission request packet to the server computer for the first missing data packet upon determining the first Round_-- Trip_-- Time is less than the first time; and

matching a transmission rate dynamically to an available bandwidth capacity of a network connection between the server computer and the client computer.

26. A computer readable medium having computer executable instructions for causing a client computer to perform a method comprising:

detecting that a first missing data packet from a data stream has not arrived at the client computer by a first expected time of arrival (ETA);

computing a first Round_-- Trip_-- Time for the first missing data packet, the first Round_-- Trip_-- Time being an estimate of a period beginning from a time a first retransmission request is sent from the client computer to a server computer till a time a copy of the first missing data packet is received at the client computer from the server computer in response to the first retransmission request;

computing a first time remaining before the first missing data packet is due for processing by an application on the client computer; and

sending a first retransmission request packet to the server computer for the first missing data packet upon determining the first Round_-- Trip_-- Time is less than the first time.

27. The computer readable medium of claim 26 wherein upon determining the first missing data packet is an important data packet, the first retransmission request packet is sent to the server computer for the first missing data packet even upon determining the first Round_-- Trip_-- Time is greater than the first time remaining before the first missing data packet is due for processing by the application on the client computer.

28. The computer readable medium of claim 26 wherein the data stream is a video stream and the first missing data packet includes an I video frame.

29. The computer readable medium of claim 26 wherein the data stream is a video stream and the first missing data packet includes a base layer video frame.

30. The computer readable medium of claim 26 wherein the data stream is an audio stream, the first missing data packet is an audio packet and a data size of the audio packet is greater than a first difference between the first Round_-- Trip_-- Time and the first time remaining before the first missing data packet is due for processing by the application on the client computer.

31. The computer readable medium of claim 26, wherein the method further comprises:

detecting that a second missing data packet has not arrived at the client computer by a second ETA;

computing a second Round_-- Trip_-- Time for the second missing data packet, the second Round_-- Trip_-- Time being an estimate of a period beginning from a time a second retransmission request is sent from the client computer to the server computer till a time a copy of the second missing data packet is received at the client computer from the server computer in response to the second retransmission request;

computing a second time remaining before the second missing data packet is due for processing by the application on the client computer;

upon determining the second Round_-- Trip_-- Time is less than the second time remaining before the second missing data packet is due for processing by the application on the client computer, sending the second retransmission request to the server for the second missing data packet prior to sending a first retransmission request packet to the server computer upon determining the second missing data packet has a higher priority than the first missing data packet; and

upon determining the second Round_-- Trip_-- Time is less than the second time remaining before the second missing data packet is due for processing by the application on the client computer, sending the second retransmission request to the server for the second missing data packet after sending the first retransmission request packet to the server computer upon determining the second missing data packet has a lower priority than the first missing data packet.

32. The computer readable medium of claim 31 wherein the priority of first and second missing data packets is based on the respective data type of the missing data packets.

33. The computer readable medium of claim 31 wherein the first missing data packet includes an I video frame, the second missing data packet includes a P video frame, and hence the first missing data packet has a higher priority than the second missing data packet.

34. The computer readable medium of claim 31 wherein the first missing data packet includes a base layer video frame, the second missing data packet includes an enhancement layer video frame, and hence the first missing data packet has a higher priority than the second missing data packet.

35. The computer readable medium of claim 31 wherein a first difference between the first Round_-- Trip_-- Time and the first time remaining is less than a second difference between the second Round_-- Trip_-- Time and the second time remaining, and hence the first missing data packet has a higher priority than the second missing data packet.

36. The computer readable medium of claim 35 wherein the data stream is a video data stream and wherein the first and the second missing data packets include video frames.

37. The computer readable medium of claim 35 wherein the data stream is an annotation stream and wherein the first and second missing data packets include annotation frames.

38. A computer readable medium having computer executable instructions for causing a client computer to perform a method comprising:

detecting that a first missing data packet has not arrived at a client computer by a first expected time of arrival (ETA);

computing a first Round_-- Trip_-- Time for the first missing data packet, the first Round_-- Trip_-- Time being an estimate of a period beginning from a time a first retransmission request is sent from the client computer to a server computer till a time a copy of the first missing data packet is received at the client computer from the server computer in response to the first retransmission request;

computing a first time remaining before the first missing data packet is due for processing by an application on the client computer;

sending a first retransmission request packet to the server computer for the first missing data packet upon determining the first Round_-- Trip_-- Time is less than the first time; and

matching a transmission rate dynamically to an available bandwidth capacity of a network connection between the server computer and the client computer.