Flow control method for quality streaming of audio/video/media over packet networks

• US 20030067872A1
• Filed: 09/13/2002
• Published: 04/10/2003
• Est. Priority Date: 09/17/2001
• Status: Active Grant

• Current Assignee: Altera Corporation (Intel Corporation)
• Original Assignee: Pulsent Corp.
• Inventors: Prakash, Adityo, Ratner, Edward R., Miller, Thomas D., So, Hon Hing, Harrell, Chandlee
• Granted Patent: US 7,274,661 B2
• Time in Patent Office: Days
• US Class Current: 370/229
• CPC Class Codes: H04L 47/10 Flow control; Congestion control
• CPC Class Codes: H04L 47/12 Avoiding congestion; Recovering from congestion
• CPC Class Codes: H04L 47/18 End to end
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/612 for unicast
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 65/752 adapting media to network capabilities
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04N 21/23406 involving management of server-side video buffer
• CPC Class Codes: H04N 21/2343 involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/23611 Insertion of stuffing data into a multiplex stream, e.g. to obtain a constant bitrate synchronisation arrangements in time-division multiplex systems using bit stuffing for systems with different or fluctuating information rates H04J3/073
• CPC Class Codes: H04N 21/23805 Controlling the feeding rate to the network, e.g. by controlling the video pump
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/44004 involving video buffer management, e.g. video decoder buffer or video display buffer
• CPC Class Codes: H04N 21/44209 Monitoring of downstream path of the transmission network originating from a server, e.g. bandwidth variations of a wireless network arrangements for maintenance or administration in data switching networks involving bandwidth and capacity management H04L41/0896
• CPC Class Codes: H04N 21/4424 Monitoring of the internal components or processes of the client device, e.g. CPU or memory load, processing speed, timer, counter or percentage of the hard disk space used error monitoring in general G06F11/30; arrangements for monitoring conditions of receiving stations in broadcast systems H04H60/32; diagnosis, testing or measuring for television receivers H04N17/04
• CPC Class Codes: H04N 21/47202 for requesting content on demand, e.g. video on demand
• CPC Class Codes: H04N 21/6125 involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 21/6377 directed to server one-way streaming services wherein the source is controlled by the destination H04L65/613
• CPC Class Codes: H04N 21/6379 directed to encoder , e.g. for requesting a lower encoding rate
• CPC Class Codes: H04N 21/658 : Transmission by the client directed to the server

Abstract

A method and apparatus for client-side detection of network congestion in a best-effort packet network comprising streaming media traffic is disclosed. Said method and apparatus provide for quality streaming media services in a congested network with constrained bandwidth over the last-mile link. A client media buffer detects at least one level of congestion and signals a server to enact at least one error mechanism. Preferred error mechanisms include packet retransmissions, stream prioritization, stream acceleration, changes in media compression rate, and changes in media resolution. Said method and apparatus allow distributed management of network congestion for networks comprising multiple clients and carrying significant streaming media traffic.

Claims

1. A method for client-based detection of network congestion for media streaming over best-effort packet networks.

3. The method of claim 1, wherein said best-effort packet network comprises at least one server and at least one client coupled to the at least one server via a link, and wherein said at least one client receives streaming media from the at least one server, said method comprising:

buffering the media stream at the client side; and

monitoring fluctuations in the buffer level to detect network congestion.

4. The method of claim 3, wherein said link between the at least one server and at least one client is constrained such that little headroom remains above the bandwidth devoted to the media stream.

5. The method of claim 3, wherein the client monitoring of fluctuations in the buffer level further comprises:

recognizing the encoded bit rate of the current media stream;

recognizing the bandwidth constraint for the client;

measuring the current depth of the buffer;

keeping track of missing or uncorrected packets in the buffer; and

using the above data to calculate whether there is sufficient time to continue delivering the media stream at the current encoded bit rate within the current bandwidth constraint.

6. The method of claim 3, wherein at least one watermark in the client media buffer corresponding to the relative presentation time of the media stream is used to detect at least one stage of alert with respect to the point at which not enough time will remain to receive the current media stream at the current bandwidth without experiencing data loss and thus playback error.

7. The method of claim 3, wherein the client sends signals instructing the server to initiate at least one error avoidance mechanism when congestion is detected.

8. The method of claim 7, wherein the client is able to send a plurality of signals instructing the server to initiate a plurality of mechanisms for error avoidance during a plurality of statistically less frequent but more severe congestion situations.

9. The method of claim 7, wherein a mechanism for error avoidance comprises retransmitting missing and corrupted packets to the client.

10. The method of claim 7, wherein a mechanism for error avoidance comprises increasing the metering rate for a given stream.

11. The method of claim 7, wherein a mechanism for error avoidance comprises raising the server'"'"'s kernel priority for a given stream.

12. The method of claim 7, wherein a mechanism for error avoidance comprises increasing the DiffServ priority associated to a given stream.

13. The method of claim 7, wherein a mechanism for error avoidance comprises switching a given stream to a stream encoded at a lower bit rate, whereby said stream demands less bandwidth and thus allows more headroom for error correction.

14. The method of claim 7, wherein a mechanism for error avoidance comprises switching a given stream to a stream encoded at a lower bit rate and increasing the metering rate for said lower-bit-rate stream to refill the buffer.

15. The method of claim 14, further comprising switching from the lower-bit-rate stream back to the original stream when the server determines that congestion has subsided sufficiently.

16. The method of claim 15, wherein the metering rate for the lower-bit-rate stream is incremented steadily and decremented whenever the NACK rate is above a threshold, and wherein the server determines that congestion has subsided when it has served the lower-bit-rate stream at a maximum metering rate for a threshold period of time.

17. The method of claim 15, further comprising sending jamming data from the server to the client to test for congestion, wherein the jamming rate is incremented steadily and decremented whenever the NACK rate is above a threshold, and wherein the server determines that congestion has subsided when it has sent data at a maximum rate for a threshold period of time.

18. The method of claim 14, further comprising switching from the lower-bit-rate stream back to the original stream when the client determines that congestion has subsided sufficiently.

19. The method of claim 7, wherein the media stream is encoded using multi-layer coding techniques, and wherein a mechanism for error avoidance comprises scaling down the number of bits used for the stream.

20. The method of claim 7, wherein a mechanism for error avoidance comprises correcting lost bits using Forward Error Correcting codes included in the bit stream.

21. The method of claim 7, wherein each media stream begins in a start mode, said start mode comprising:

the server serving a lower-bit-rate stream to the client to fill the buffer during a buffer fill period,

whereby the buffer fills to a safe level more quickly and latency is reduced.

22. The method of claim 21, further comprising:

the client monitoring congestion during the buffer fill period and signaling the server when congestion is detected; and

the server continuing to serve the lower-bit-rate stream after playback begins if an unacceptable level of congestion is detected during the buffer fill period, or the server switching to a higher-bit-rate stream when the client begins playback if the congestion level is acceptable during the buffer fill period.

23. The method of claim 8, wherein the plurality of mechanisms for error avoidance includes at least:

retransmitting missing and corrupted packets to the client; and

switching the stream to a stream encoded at a lower bit rate to reduce the stream'"'"'s bandwidth and thus to allow more overhead for error correction.

24. The method of claim 8, wherein the plurality of mechanisms for error avoidance includes switching the stream sequentially from a high bit rate to at least two incrementally lower bit rates as needed to adjust to incrementally worse congestion situations.

25. The method of claim 7, wherein the streaming media comprises streaming video.

26. The method of claim 25, wherein the streaming video is encoded to include bi-directionally predicted frames, and wherein a mechanism for error avoidance comprises dropping said bi-directionally predicted frames from the stream and replacing them on the client side with interpolated or replicated frames.

27. The method of claim 25, wherein a mechanism for error avoidance comprises switching a given video stream to a stream encoded at a different presentation resolution and at a lower bit rate.

28. The method of claim 25, wherein a mechanism for error avoidance comprises streaming audio only when severe congestion prevents successful transmission of visual data even when other mechanisms for error avoidance are employed, whereby audio continuity is maintained in the statistically rare case when video cannot be streamed continuously.

29. The method of claim 3, wherein said link between the at least one client and the at least one server comprises an IP over DSL link.

30. The method of claim 29, wherein the streaming media comprises streaming video.

31. The method of claim 3, wherein said link between the at least one client and the at least one server comprises a wireless data network link.

32. The method of claim 3, wherein said link between the at least one client and the at least once server comprises a general IP network link.

2. A method for client-initiated error avoidance for media streaming over best-effort packet networks, wherein a client initiates at least one error avoidance mechanism in response to at least one client-detected level of network congestion.

33. A method for client-initiated flow control for media streaming over best-effort packet networks, wherein, upon client detection of network congestion, a client sends requests to a server to adjust the media stream and thereby to improve the congestion situation.

34. A method for distributed detection and control of network congestion in a best-effort packet network, said packet network comprising at least one server and a plurality of clients coupled to the at least one server via links, wherein a significant proportion of network bandwidth is devoted to streaming media traffic, said method comprising:

client-side detection of network congestion;

client-side requests for streaming media flow control for error avoidance; and

flow control measures enacted by the server in response to client requests, resulting in the aggregate in the alleviation of the network congestion.

35. The method of claim 34, wherein the client detection of network congestion comprises:

buffering the media stream at the client side; and

monitoring fluctuations in the buffer level to detect network congestion.

36. The method of claim 34, wherein client-side requests for streaming media flow control comprise a plurality of levels of requests for a plurality of flow control measures resulting in graceful degradation of quality through statistically less frequent congestion situations.

37. The method of claim 34, wherein a flow control measure comprises switching a given stream to a stream encoded at a lower bit rate, whereby said stream consumes less network bandwidth.

38. The method of claim 37, further comprising switching from the lower-bit-rate stream back to the original stream when the server determines that congestion has subsided sufficiently.

39. The method of claim 38, wherein the metering rate for the lower-bit-rate stream is incremented steadily and decremented whenever the NACK rate is above a threshold, and wherein the server determines that congestion has subsided when it has served the lower-bit-rate stream at a maximum metering rate for a threshold period of time.

40. The method of claim 38, further comprising sending jamming data from the server to the client to test for congestion, wherein the jamming rate is incremented steadily and decremented whenever the NACK rate is above a threshold, and wherein the server determines that congestion has subsided when it has sent jamming data at a maximum rate for a threshold period of time.

41. The method of claim 37, further comprising switching from the lower-bit-rate stream back to the original stream when the client determines that congestion has subsided sufficiently.

42. The method of claim 34, wherein the media stream is encoded using multi-layered coding techniques, and wherein a mechanism for error avoidance comprises scaling down the number of bits used for the stream.

43. The method of claim 34, wherein the plurality of flow control measures includes switching the stream sequentially to at least two streams encoded at incrementally lower bit rates as needed to adjust to incrementally worse congestion situations.

44. The method of claim 34, wherein the streaming media comprises streaming video.

45. The method of claim 44, wherein the streaming video is encoded to include bi-directionally predicted frames, and wherein a flow control measure comprises dropping said bi-directionally predicted frames from the stream and replacing them on the client side with interpolated or replicated frames.

46. The method of claim 44, wherein a flow control measure comprises switching a given video stream to a stream encoded at a different presentation resolution and at a lower bit rate.

47. The method of claim 44, wherein a flow control measure comprises streaming audio only when severe congestion prevents successful transmission of visual data even when other flow control measures are employed, whereby audio continuity is maintained in the statistically rare case when video cannot be streamed continuously.

48. The method of claim 34, wherein said links between the plurality of clients and the at least one server comprise at least one IP over DSL link.

49. The method of claim 48, wherein the streaming media traffic comprises streaming video.

50. The method of claim 34, wherein said links between the plurality of clients and the at least one server comprise at least one wireless data network link.

51. The method of claim 34, wherein said links between the plurality of clients and the at least once server comprise general IP network links.

52. A method for detection and control of network congestion in a best-effort packet network, said packet network comprising at least one server and a plurality of clients coupled to the at least one server via links, wherein a significant proportion of network bandwidth is devoted to streaming media traffic, said method comprising:

the clients detecting network congestion via monitoring buffer levels in a client media buffer;

the clients signaling their buffer levels to the at least on server;

the server aggregating client buffer level information; and

the server initiating flow control measures to reduce media traffic and thereby to prevent more serious congestion from developing.

53. An apparatus for detecting network congestion in a best-effort packet network, said packet network comprising at least one server and at least one client coupled to the at least one server via a link, said apparatus comprising:

a client media buffer at the at least one client;

a plurality of watermarks dividing said buffer into a plurality of regions depending on a presentation time for data remaining in the buffer; and

a signaling device operatively coupled to the buffer by which said buffer sends signals to the at least one server to enact an error avoidance mechanism when said buffer operates in at least one of the regions.

54. The apparatus of claim 53, wherein the client media buffer operates in two modes.

55. The apparatus of claim 54, wherein a first mode of the two modes comprises:

an active zone including a resume water mark;

an overflow alert zone above the active zone and separated from the active zone by a high water mark;

a prioritized recovery zone below the active zone and separated from the active zone by a low water mark; and

a stream switch zone below the prioritized recovery zone and separated from the prioritized recovery zone by an ultra low water mark.

56. The apparatus of claim 55, wherein a second mode of the two modes comprises:

a fill zone;

a switch test zone above the fill zone and separated from the fill zone by a resume mark; and

an overflow alert zone above the switch test zone and separated from the switch test zone by a high water mark.

57. The apparatus of claim 56, wherein said signals sent by the signaling device to the server depend upon the zone in which the client media buffer is operating.

58. The apparatus of claim 57, wherein when the overflow alert zone is reached in either said mode, then a pause signal is sent from the signaling device to the server to indicate that serving to that client media buffer is to be paused.

59. The apparatus of claim 58, wherein when the resume mark is crossed after having entered the overflow alert zone, then a resume signal is sent from the signaling device to the server to indicate that serving to that client media buffer is to be resumed.

60. The apparatus of claim 57, wherein when the prioritized recover zone is reached, then an increase priority signal is sent from the signaling device to the server to indicate that serving priority to that client media buffer is to be increased.

61. The apparatus of claim 60, wherein when the resume mark is crossed after having entered the prioritized recovery zone, then a resume signal is sent from the signaling device to the server to indicate that serving priority to that client media buffer is to be decreased.

62. The apparatus of claim 57, wherein when the stream switch zone is reached, then the signaling device sends a switch signal to the server to indicate to the server to begin transmitting to the client media buffer a stream encoded at a lower bit rate.

63. The apparatus of claim 62, wherein when the signaling device sends the switch signal to the server, the client media buffer enters the second mode in the fill zone.

64. The apparatus of claim 57, wherein when the switch test zone is reached, then the signaling device sends a switch test signal to the server to indicate to the server that it should begin to test conditions for switching from a lower-bit-rate stream to a higher-bit-rate stream.

65. A system for distributed detection and control of network congestion in a best-effort packet network, said packet network comprising at least one server and a plurality of clients coupled to the at least one server via links, wherein a significant proportion of network bandwidth is devoted to streaming media traffic, said system comprising:

client-side means for detection of network congestion;

client-side means for signaling the server to initiate flow control measures; and

server-side means to enact flow control measures in response to client requests, resulting in the aggregate in the alleviation of the network congestion.