Adaptive bitrate management for streaming media over packet networks

• US 8,769,141 B2
• Filed: 08/28/2012
• Issued: 07/01/2014
• Est. Priority Date: 07/10/2007
• Status: Active Grant

• Current Assignee: OptiMorphix, Inc.
• Original Assignee: Citrix Systems, Inc. (Cloud Software Group)
• Inventors: Melnyk, Miguel A., Stavrakos, Nicholas J., Penner, Andrew, Tidemann, Jeremy, Breg, Fabian
• Primary Examiner(s): Chan, Wing F
• Assistant Examiner(s): MIAH, RAZU A
• Application Number:
  US13/596,916
• Publication Number:
  US 20130086275A1
• Time in Patent Office: 672 Days
• Field of Search: 709230-231, 370228-339
• US Class Current: 709/231
• CPC Class Codes: H04L 47/10 Flow control; Congestion control
• CPC Class Codes: H04L 47/193 at the transport layer, e.g. TCP related
• CPC Class Codes: H04L 47/2416 Real-time traffic
• CPC Class Codes: H04L 47/25 with rate being modified by the source upon detecting a change of network conditions
• CPC Class Codes: H04L 47/283 in response to processing delays, e.g. caused by jitter or round trip time [RTT]
• CPC Class Codes: H04L 47/32 by discarding or delaying data units, e.g. packets or frames
• CPC Class Codes: H04L 47/38 by adapting coding or compression rate
• CPC Class Codes: H04L 65/65 Network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP]
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04L 69/163 In-band adaptation of TCP data exchange; In-band control procedures
• CPC Class Codes: H04N 21/23406 involving management of server-side video buffer
• CPC Class Codes: H04N 21/234381 by altering the temporal resolution, e.g. decreasing the frame rate by frame skipping
• CPC Class Codes: H04N 21/2368 Multiplexing of audio and video streams
• CPC Class Codes: H04N 21/2385 Channel allocation H04N21/266 takes precedence; Bandwidth allocation H04N21/24 takes precedence ; allocation of channels according to the instantaneous demands of the users in time-division multiplex systems H04J3/1682; admission control, resource allocation in open networks H04L12/5692; arrangements for maintenance or administration in data switching networks involving bandwidth and capacity management H04L41/0896; negotiating bandwidth in wireless networks H04W28/16
• CPC Class Codes: H04N 21/2402 Monitoring of the downstream path of the transmission network, e.g. bandwidth available traffic monitoring in data switching networks H04L43/00; monitoring data switching networks utilization H04L43/0876
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/44004 involving video buffer management, e.g. video decoder buffer or video display buffer
• CPC Class Codes: H04N 21/6377 directed to server one-way streaming services wherein the source is controlled by the destination H04L65/613
• CPC Class Codes: H04N 21/658 Transmission by the client directed to the server
• CPC Class Codes: H04N 21/6582 : Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number arrangements where receivers interact with the broadcast H04H20/38
• CPC Class Codes: H04N 21/6583 : Acknowledgement

Abstract

A method including providing pseudo-streaming media data to a terminal; receiving a transport control protocol (TCP) acknowledgement from the terminal; estimating one or more network conditions of a network based at least in part on the TCP acknowledgement; determining an optimal session bitrate based on the estimated one or more network conditions; and providing pseudo-streaming media data to the terminal based on the optimal session bitrate.

Claims

1. A method comprising:

providing pseudo-streaming media data for transmission to a terminal, wherein the pseudo-streaming media data includes one or more sequence numbers and corresponds to one or more timestamps;

causing a table to store an association between the one or more sequence numbers and the one or more timestamps;

receiving one or more transport control protocol (TCP) acknowledgments from a terminal, wherein a TCP acknowledgment of the one or more received TCP acknowledgments includes a sequence number of the one or more sequence numbers and corresponds to a certain time;

acquiring the timestamp from the table using the sequence number of the TCP acknowledgement;

estimating one or more network conditions of a media network based at least in part on a comparison between the certain time and the acquired timestamp;

determining an optimal audio bitrate and an optimal video bitrate using the estimated one or more network conditions;

receiving media data that includes audio media data and video media data;

encoding the audio media data using the optimal audio bitrate;

encoding the video media data using the optimal video bitrate; and

providing the encoded audio media data and the encoded video media data for transmission to the terminal.

2. The method of claim 1, further comprising dropping frames of the encoded video data.

3. A method comprising:

receiving media data that includes audio media data and video media data;

receiving one or more transport control protocol (TCP) acknowledgments from a terminal;

estimating one or more network conditions of a media network using the one or more TCP acknowledgments;

determining an optimal session bitrate using the estimated one or more network conditions;

allocating the optimal session bitrate between the audio media data and the video media data to produce the optimal audio bitrate and the optimal video bitrate;

encoding the audio media data using the optimal audio bitrate;

encoding the video media data using the optimal video bitrate; and

providing the encoded audio media data and the encoded video media data for transmission to the terminal.

4. The method of claim 3, wherein allocating the optimal session bitrate between the audio media data and the video media data is based on one or more metrics selected from a group including a predetermined allocation, a user preference, an optimal performance data, privileging one type of media data over the other, and an amount of audio and video media data to be provided.

5. The method of claim 3, wherein determining the optimal session bitrate further comprises:

establishing stability criterion based on the estimated one or more network conditions;

determining the stability of the media network; and

providing the optimal session bitrate based on the stability determination.

6. The method of claim 5, wherein establishing stability criterion further comprises:

comparing a media time in transit and a round trip time estimate for assisting with the stability determination.

7. The method of claim 5, wherein establishing stability criterion further comprises:

comparing a bitrate received with a current bitrate session.

8. The method of claim 5, further comprising:

if the stability of the media network is considered normal, maintaining or incrementally increasing a current bitrate; and

if the stability of the media network is not normal, adjusting the current bitrate.

9. A terminal including one or more processors and a memory, the terminal comprising:

a buffer configured to receive pseudo-streaming media data packets provided by an adaptive bitrate manager over a network; and

a media player configured to receive pseudo-streaming media packets and to provide one or more transport control protocol (TCP) acknowledgments to the adaptive bitrate manager, wherein the adaptive bitrate manager is configured to provide the pseudo-streaming media packets to the terminal, wherein the pseudo-streaming media data includes one or more sequence numbers and corresponds to one or more timestamps, to cause a table to store an association between the one or more sequence numbers and the one or more timestamps, to receive the one or more TCP acknowledgments, wherein a TCP acknowledgment of the one or more received TCP acknowledgments includes a sequence number of the one or more sequence numbers and corresponds to a certain time, to acquire the timestamp from the table using the sequence number of the TCP acknowledgement, to estimate one or more network conditions of the media network based at least in part on a comparison between the certain time and the acquired timestamp, to determine an optimal audio bitrate and an optimal video bitrate using the estimated one or more network conditions, to receive media data that includes audio media data and video media data, to encode the audio media data using the optimal audio bitrate, to encode the video media data using the optimal video bitrate, and to provide the encoded audio media data and the encoded video media data as media data packets to the terminal.

10. A system including one or more processors and a memory, the system comprising:

an adaptive bitrate manager configured to estimate one or more network conditions of a media network between the adaptive bitrate manager and a terminal using one or more received TCP acknowledgments and to acquire media data that includes audio media data and video media data, the adaptive bitrate manager further comprising;

an adaptive bitrate controller configured to determine an optimal session bitrate using the estimated one or more network conditions,

a bitrate splitter configured to acquire the optimal session bitrate and allocate the optimal session bitrate between the audio media data and the video media data to produce an optimal audio bitrate and an optimal video bitrate, and

one or more encoders configured to;

encode the audio media data using the optimal audio bitrate,

encode the video media data using the optimal video bitrate, and

provide the encoded audio media data and the encoded video media data for transmission to the terminal.

11. A system including one or more processors and a memory, the system comprising:

an adaptive bitrate manager configured to provide pseudo-streaming media data for transmission to a terminal, wherein the provided pseudo-streaming media data includes one or more sequence numbers and corresponds to one or more timestamps, to cause a table to store an association between the one or more sequence numbers and the one or more timestamps, to receive one or more TCP acknowledgments, wherein a TCP acknowledgment of the one or more received TCP acknowledgments includes a sequence number of the one or more sequence numbers and corresponds to a certain time, to acquire the timestamp from the table using the sequence number of the TCP acknowledgment, to estimate one or more network conditions of a media network based at least in part on a comparison between the certain time and the acquired timestamp, and to acquire media data that includes audio media data and video media data, the adaptive bitrate manager further comprising;

a bitrate splitter configured to determine an optimal audio bitrate and an optimal video bitrate using the estimated one or more network conditions, and

one or more encoders configured to;

encode the audio media data using the optimal audio bitrate,

encode the video media data using the optimal video bitrate, and

provide the encoded audio media data and the encoded video media data for transmission to the terminal.

12. A non-transitory computer readable medium storing instructions that, when executed by an adaptive bitrate manager including one or more processors, cause the adaptive bitrate manager to perform a method for processing one or more TCP acknowledgments, the method comprising:

providing pseudo-streaming media data for transmission to a terminal, wherein the provided pseudo-streaming media data includes one or more sequence numbers and corresponds to one or more timestamps;

causing a table to store an association between the one or more sequence numbers and the one or more timestamps;

receiving one or more transport control protocol (TCP) acknowledgments from a terminal, wherein a TCP acknowledgment of the one or more received TCP acknowledgments includes a sequence number of the one or more sequence numbers and corresponds to a certain time;

acquiring the timestamp from the table using the sequence number of the TCP acknowledgement;

estimating one or more network conditions of a media network based at least in part on a comparison between the certain time and the acquired timestamp;

determining an optimal audio bitrate and an optimal video bitrate using the estimated one or more network conditions;

receiving media data that includes audio media data and video media data;

encoding the audio media data using the optimal audio bitrate;

encoding the video media data using the optimal video bitrate; and

providing the encoded audio media data and the encoded video media data for transmission to the terminal.

13. The non-transitory computer readable medium of claim 12, further comprising instructions that, when executed by an adaptive bitrate manager including one or more processors, cause the adaptive bitrate manager to perform:

dropping frames of the encoded video data.

14. A non-transitory computer readable medium storing instructions that, when executed by an adaptive bitrate manager including one or more processors, cause the adaptive bitrate manager to perform a method for processing one or more TCP acknowledgments, the method comprising:

receiving media data that includes audio media data and video media data;

receiving one or more transport control protocol (TCP) acknowledgments from a terminal;

estimating one or more network conditions of a media network using the one or more TCP acknowledgments;

determining an optimal session bitrate using the estimated one or more network conditions;

allocating the optimal session bitrate between the audio media data and the video media data to produce the optimal audio bitrate and the optimal video bitrate;

encoding the audio media data using the optimal audio bitrate;

encoding the video media data using the optimal video bitrate; and

providing the encoded audio media data and the encoded video media data for transmission to the terminal.

15. The non-transitory computer readable medium of claim 14, wherein allocating the optimal session bitrate between the audio media data and the video media data is based on one or more metrics selected from a group including a predetermined allocation, a user preference, an optimal performance data, privileging one type of media data over the other, and an amount of audio and video media data to be provided.

16. The non-transitory computer readable medium of claim 14, wherein determining the optimal session bitrate further comprises:

establishing stability criterion based on the estimated one or more network conditions;

determining the stability of the media network; and

providing the optimal session bitrate based on the stability determination.

17. The non-transitory computer readable medium of claim 16, wherein establishing stability criterion further comprises:

comparing a media time in transit and a round trip time estimate for assisting with the stability determination.

18. The non-transitory computer readable medium of claim 16, wherein establishing stability criterion further comprises:

comparing a bitrate received with a current bitrate session.

19. The non-transitory computer readable medium of claim 16, further comprising:

if the stability of the media network is considered normal, maintaining or incrementally increasing a current bitrate; and

if the stability of the media network is not normal, adjusting the current bitrate.

20. A method comprising:

receiving media data that includes audio media data and video media data;

receiving an optimal session bitrate;

allocating the optimal session bitrate between the audio media data and the video media data to produce an optimal audio bitrate and an optimal video bitrate, wherein allocating the optimal session bitrate between audio and video media can involves allocating a higher bitrate for either the audio media or the video media over the other;

encoding the audio media data using the optimal audio bitrate;

encoding the video media data using the optimal video bitrate;

multiplexing the encoded audio media data and the encoded video media data; and

providing the multiplexed encoded audio and video media data for transmittal to a terminal.

21. A non-transitory computer readable storage medium storing instructions that, when executed by one or more computers, cause the one or more computers to perform a method for processing an optimal session bitrate, the method comprising:

receiving media data that includes audio media data and video media data;

receiving the optimal session bitrate;

allocating the optimal session bitrate between the audio media data and the video media data to produce an optimal audio bitrate and an optimal video bitrate, wherein allocating the optimal session bitrate between audio and video media can involves allocating a higher bitrate for either the audio media or the video media over the other;

encoding the audio media data using the optimal audio bitrate;

encoding the video media data using the optimal audio bitrate and the optimal video bitrate;

multiplexing the encoded audio media data and the encoded video media data; and

providing the multiplexed encoded audio and video media data for transmittal to a terminal.

22. A terminal including one or more processors and a memory, the terminal comprising:

a buffer configured to receive pseudo-streaming media data packets provided by an adaptive bitrate manager over a network; and

a media player configured to receive pseudo-streaming media packets and to provide one or more transport control protocol (TCP) acknowledgments to the adaptive bitrate manager, wherein the adaptive bitrate manager is configured to receive media data that includes audio media data and video media data, to receive the one or more TCP acknowledgments, to estimate one or more network conditions of the media network using the one or more TCP acknowledgments, to determine an optimal session bitrate using the estimated one or more network conditions, to allocate the optimal session bitrate between the audio media data and the video media data to produce an optimal audio bitrate and an optimal video bitrate, to encode the audio media data using the optimal audio bitrate, to encode the video media data using the optimal video bitrate, and to provide the encoded audio media data and the encoded video media data for transmission as media data packets to the terminal.