Video rate adaptation to reverse link conditions

• US 8,406,309 B2
• Filed: 05/31/2006
• Issued: 03/26/2013
• Est. Priority Date: 10/21/2005
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Lee, Yen-Chi, Lott, Christopher Gerard, Tinnakornsrisuphap, Peerapol, Gupta, Vikram
• Primary Examiner(s): Rao, Andy
• Application Number:
  US11/445,099
• Publication Number:
  US 20090034610A1
• Time in Patent Office: 2,491 Days
• Field of Search: 37524001-24029
• US Class Current: 375/240.26
• CPC Class Codes: H04N 19/115 Selection of the code volume for a coding unit prior to coding
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/147 according to rate distortion criteria rate-distortion as a criterion for motion estimation H04N19/567
• CPC Class Codes: H04N 19/152 by measuring the fullness of the transmission buffer
• CPC Class Codes: H04N 19/164 Feedback from the receiver or from the transmission channel
• CPC Class Codes: H04N 19/169 characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/184 the unit being bits, e.g. of the compressed video stream
• CPC Class Codes: H04N 19/188 the unit being a video data packet, e.g. a network abstraction layer [NAL] unit
• CPC Class Codes: H04N 19/194 involving only two passes
• CPC Class Codes: H04N 19/196 being specially adapted for the computation of encoding parameters, e.g. by averaging previously computed encoding parameters processing of motion vectors H04N19/513
• CPC Class Codes: H04N 19/198 including smoothing of a sequence of encoding parameters, e.g. by averaging, by choice of the maximum, minimum or median value
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 21/23406 involving management of server-side video buffer
• CPC Class Codes: H04N 21/2368 Multiplexing of audio and video streams
• CPC Class Codes: H04N 21/23805 Controlling the feeding rate to the network, e.g. by controlling the video pump
• CPC Class Codes: H04N 21/2381 Adapting the multiplex stream to a specific network, e.g. an Internet Protocol [IP] network transmission of MPEG streams over ATM H04L12/5601
• CPC Class Codes: H04N 21/2402 Monitoring of the downstream path of the transmission network, e.g. bandwidth available traffic monitoring in data switching networks H04L43/00; monitoring data switching networks utilization H04L43/0876
• CPC Class Codes: H04N 21/2662 : Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/4341 : Demultiplexing of audio and video streams
• CPC Class Codes: H04N 21/44004 : involving video buffer management, e.g. video decoder buffer or video display buffer
• CPC Class Codes: H04N 21/6125 : involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 21/6131 : involving transmission via a mobile phone network wireless downlink channel access H04W74/006
• CPC Class Codes: H04N 21/6175 : involving transmission via Internet broadcast-related systems characterised by the transmission system being the Internet H04H60/82
• CPC Class Codes: H04N 21/6377 : directed to server one-way streaming services wherein the source is controlled by the destination H04L65/613
• CPC Class Codes: H04N 21/64322 : IP
• CPC Class Codes: H04N 21/6437 : Real-time Transport Protocol [RTP]
• CPC Class Codes: H04N 21/64769 : for rate control flow control in packet networks H04L47/10
• CPC Class Codes: H04N 21/64792 : Controlling the complexity of the content stream, e.g. by dropping packets intermediate media network packet handling H04L65/765; proxy provisioning conversion or adaptation for reducing the amount or size of exchanged application data H04L67/5651; negotiation of resources in wireless networks H04W28/16
• CPC Class Codes: H04N 21/658 : Transmission by the client directed to the server
• CPC Class Codes: H04W 52/04 : TPC

Abstract

The disclosure relates to video rate adaptation techniques that may use information from a medium access control (MAC) layer and radio link protocol (RLP) layer. The techniques may greatly reduce video delay by adjusting video encoding rate. For real-time video telephony (VT) applications, these techniques may provide graceful quality degradation and improve user experience, especially when the channel conditions degrade.

Claims

1. A method of video encoding comprising:

estimating, via an estimation unit of an encoder system, video throughput of a transmission channel based on a size of a video flow radio link protocol (RLP) queue at an access terminal, wherein a transmission rate of data across the transmission channel varies; and

encoding, via an encoder of the encoder system, video data using the estimated video throughput.

2. A method of controlling a video encoding rate, the method comprising:

determining, via an estimation unit of an encoder system, a first size V_n of a video queue in a radio link protocol (RLP) layer at a first time t_n based on a video frame rate;

determining, via the estimation unit, a second size V_m of the video queue at a second time t_m based on an audio frame rate;

if the first size V_n or the second size V_m is greater than zero, then using the first size V_n, a previous size V_n-1 of the video queue associated with a previous video frame, a previous video frame size B_n-1, the first time t_n, and a time t_n-1 associated with the previous size of the video queue to determine an estimated video throughput VTP of a transmission channel;

if the first size V_n and the second size V_m are equal to zero, then searching for an earlier time based on the audio frame rate when the video queue size was greater than zero;

after finding the earlier time based on the audio frame rate when the video queue size was greater than zero, using an earlier queue size V_m-i based on the audio frame rate, the previous size V_n-1 of the video queue associated with the previous video frame, the previous video frame size B_n-1, the earlier time t_m-1, and the time t_n-1 associated with the previous size of the video queue to determine the estimated video throughput VTP of the transmission channel;

using the estimated video throughput VTP to determine a channel-constrained video frame size; and

using the channel-constrained video frame size to control a video encoding rate.

3. The method of claim 2, wherein using the first size (V_x), a previous size (V_n-1) of the video queue associated with a previous video frame, a previous video frame size (B_n-1), the first time (t_x), and a time (t_n-1) associated with the previous size of the video queue to determine the estimated video throughput (VTP) comprises:

4. The method of claim 2, wherein using the earlier size (V_x), the previous size (V_n-1) of the video queue associated with the previous video frame, the previous video frame size (B_n-1), the earlier time (t_x), and the time (t_n-1) associated with the previous size of the video queue to determine the estimated video throughput (VTP) comprises:

5. The method of claim 2, further comprising if the first size and the second size are equal to zero, and a third size of the video queue associated with a previous audio frame is greater than zero, then using the third size, the previous size of the video queue associated with a previous video frame, the previous video frame size, a time associated with the third size, and the time associated with the previous size of the video queue to determine the estimated video throughput.

6. The method of claim 2, wherein the video queue is assigned to only store video packets.

7. The method of claim 2, wherein using the estimated video throughput to determine a channel-constrained video frame size further uses the video frame rate, the first size and an adjustment factor, which controls how much the video encoding rate reacts to channel conditions, wherein a video frame size is determined based on the video frame rate and the estimated video throughput is increased by the adjustment factor.

8. The method of claim 7, wherein the adjustment factor has a first value for video conferencing and a second value for video sharing.

9. The method of claim 2, further comprising:

using the video encoding rate to encode video; and

transmitting the encoded video across a wireless channel using Code Division Multiple Access Evolution Data Optimized Revision A (CDMA EV-DO Rev A) protocols.

10. The method of claim 2, further comprising:

determining whether the channel-constrained video frame size is greater than a minimum frame size threshold;

if the channel-constrained video frame size is greater than the minimum frame size threshold, determining whether the channel-constrained video frame size is smaller than a frame size constrained by a virtual buffer;

if the channel-constrained video frame size is smaller than the frame size constrained by the virtual buffer, then using the channel-constrained video frame size to control the video encoding rate; and

if the channel-constrained video frame size is greater than the frame size constrained by the virtual buffer, then using the frame size constrained by the virtual buffer to control the video encoding rate.

11. A method comprising:

determining, via an estimation unit of an encoder system, a size of a video queue in a radio link protocol (RLP) layer;

determining, via the estimation unit, a transmit power headroom limitation from a medium access control (MAC) layer;

using the determined transmit power headroom limitation to determine a MAC payload size;

using the determined MAC payload size and an estimate of how many transmission opportunities are given to video in a time period to determine an estimated video throughput;

using the estimated video throughput and the determined size of the video queue in the RLP layer to determine a channel-constrained video frame size; and

using the channel-constrained video frame size to control a video encoding rate.

12. The method of claim 11, wherein using the estimated video throughput and the determined size of the video queue in the RLP layer to determine a channel-constrained video frame size further uses an adjustment factor, which controls how much the video encoding rate reacts to channel conditions.

13. The method of claim 12, wherein the adjustment factor has a first value for video conferencing and a second value for video sharing.

14. The method of claim 11, further comprising:

determining whether the channel-constrained video frame size is greater than a minimum frame size threshold;

if the channel-constrained video frame size is greater than the minimum frame size threshold, determining whether the channel-constrained video frame size is smaller than a frame size constrained by a virtual buffer;

if the channel-constrained video frame size is smaller than the frame size constrained by the virtual buffer, then using the channel-constrained video frame size to control the video encoding rate; and

if the channel-constrained video frame size is greater than the frame size constrained by the virtual buffer, then using the frame size constrained by the virtual buffer to control the video encoding rate.

15. The method of claim 11, further comprising:

using the video encoding rate to encode video; and

transmitting the encoded video across a wireless channel using Code Division Multiple Access Evolution Data Optimized Revision A (CDMA EV-DO Rev A) protocols.

16. The method of claim 11, wherein the video queue is assigned to only store video packets.

17. An apparatus that encodes video data, the apparatus comprising a processor and a machine-readable memory storing a set of instructions that, when executed by the processor, cause the apparatus to:

determine a first size V_n of a video queue in a radio link protocol (RLP) layer at a first time t_n based on a video frame rate;

determine a second size V_m of the video queue at a second time t_m based on an audio frame rate;

if the first size V_n or the second size V_m is greater than zero, then use the first size V_n, a previous size V_n-1 of the video queue associated with a previous video frame, a previous video frame size B_n-1, the first time tn, and a time t_n-1 associated with the previous size of the video queue to determine an estimated video throughput VTP of a transmission channel;

if the first size V_n and the second size V_m are equal to zero, then searching for an earlier time based on the audio frame rate when the video queue size was greater than zero;

after finding the earlier time based on the audio frame rate when the video queue size was greater than zero, use an earlier queue size V_m-i based on the audio frame rate, the previous size V_n-1 of the video queue associated with the previous video frame, the previous video frame size B_n-1, the earlier time t_m-1, and the time t_n-1 associated with the previous size of the video queue to determine the estimated video throughput VTP of the transmission channel;

use the estimated video throughput VTP to determine a channel-constrained video frame size; and

use the channel-constrained video frame size to control a video encoding rate.

18. The apparatus of claim 17, wherein instructions operate on an application layer above the RLP layer.

19. The apparatus of claim 17, wherein a medium access control (MAC) layer for video flows of the apparatus does not react to sector loading.

20. The apparatus of claim 17, wherein the instructions that, when executed by the processor, cause the apparatus to use the first size (V_x), a previous size (V_n-1) of the video queue associated with a previous video frame, a previous video frame size (B_n-1), the first time (t_x), and a time (t_n-1) associated with the previous size of the video queue to determine the estimated video throughput (VTP) comprise:

21. The apparatus of claim 17, wherein the instructions that, when executed by the processor, cause the apparatus to use the earlier size (V_x), the previous size (V_n-1) of the video queue associated with the previous video frame, the previous video frame size (B_n-1), the earlier time (t_x), and the time (t_n-1) associated with the previous size of the video queue to determine the estimated video throughput (VTP) comprise:

22. The apparatus of claim 17, wherein the instructions further comprise if the first size and the second size are equal to zero, and a third size of the video queue associated with a previous audio frame is greater than zero, then using the third size, the previous size of the video queue associated with a previous video frame, the previous video frame size, a time associated with the third size, and the time associated with the previous size of the video queue to determine the estimated video throughput.

23. The apparatus of claim 17, wherein the video queue is assigned to only store video packets.

24. The apparatus of claim 17, wherein the instructions that, when executed by the processor, cause the apparatus to use the estimated video throughput to determine a channel-constrained video frame size further comprise instructions that, when executed by the processor, cause the apparatus to use the video frame rate, the first size and an adjustment factor, which controls how much the video encoding rate reacts to channel conditions, wherein a video frame size is determined based on the video frame rate and the estimated video throughput is increased by the adjustment factor.

25. The apparatus of claim 24, wherein the adjustment factor has a first value for video conferencing and a second value for video sharing.

26. The apparatus of claim 17, further comprising:

a video encoder configured to use the video encoding rate to encode video; and

a transmitter configured to transmit the encoded video across a wireless channel using Code Division Multiple Access Evolution Data Optimized Revision A (CDMA EV-DO Rev A) protocols.

27. The apparatus of claim 17, further comprising instructions that, when executed by the processor, cause the apparatus to:

determine whether the channel-constrained video frame size is greater than a minimum frame size threshold;

if the channel-constrained video frame size is greater than the minimum frame size threshold, determine whether the channel-constrained video frame size is smaller than a frame size constrained by a virtual buffer;

if the channel-constrained video frame size is smaller than the frame size constrained by the virtual buffer, then use the channel-constrained video frame size to control the video encoding rate; and

if the channel-constrained video frame size is greater than the frame size constrained by the virtual buffer, then use the frame size constrained by the virtual buffer to control the video encoding rate.

28. An apparatus that encodes video data, the apparatus comprising a processor and a machine-readable memory storing a set of instructions that, when executed by the processor, cause the apparatus to:

determine a size of a video queue in a radio link protocol (RLP) layer;

determine a transmit power headroom limitation from a medium access control (MAC) layer;

use the determined transmit power headroom limitation to determine a MAC payload size;

use the determined MAC payload size and an estimate of how many transmission opportunities are given to video in a time period to determine an estimated video throughput;

use the estimated video throughput and the determined size of the video queue in the RLP layer to determine a channel-constrained video frame size; and

use the channel-constrained video frame size to control a video encoding rate.

29. The apparatus of claim 28, wherein the instructions that, when executed by the processor, cause the apparatus to use the estimated video throughput and the determined size of the video queue in the RLP layer to determine a channel-constrained video frame size further comprise instructions that, when executed by the processor, use an adjustment factor, which controls how much the video encoding rate reacts to channel conditions.

30. The apparatus of claim 28, further comprising instructions that when executed by the processor, cause the apparatus to:

determine whether the channel-constrained video frame size is greater than a minimum frame size threshold;

if the channel-constrained video frame size is greater than the minimum frame size threshold, determine whether the channel-constrained video frame size is smaller than a frame size constrained by a virtual buffer;

if the channel-constrained video frame size is smaller than the frame size constrained by the virtual buffer, then use the channel-constrained video frame size to control the video encoding rate; and

if the channel-constrained video frame size is greater than the frame size constrained by the virtual buffer, then use the frame size constrained by the virtual buffer to control the video encoding rate.

31. An apparatus that encodes video data, the apparatus comprising:

a radio link protocol (RLP) layer queue configured to store video data;

a first unit configured to receive a size of the RLP video queue and a transmit power headroom limitation from a medium access control (MAC) layer, use the transmit power headroom limitation to determine a MAC payload size, use the determined MAC payload size and an estimate of how many transmission opportunities are given to video in a time period to determine video throughput, and use the determined video throughput and the size of the video queue in the RLP layer to determine a channel-constrained video frame size;

a second unit to use the channel-constrained video frame size to control a video encoding rate; and

a video encoder to use the video encoding rate to encode video.

32. The apparatus of claim 31, further comprising:

a virtual buffer; and

a third unit configured to;

determine whether the channel-constrained video frame size is greater than a minimum frame size threshold;

if the channel-constrained video frame size is greater than the minimum frame size threshold, determine whether the channel-constrained video frame size is smaller than a frame size constrained by a virtual buffer;

if the channel-constrained video frame size is smaller than the frame size constrained by the virtual buffer, then use the channel-constrained video frame size to control the video encoding rate; and

if the channel-constrained video frame size is greater than the frame size constrained by the virtual buffer, then use the frame size constrained by the virtual buffer to control the video encoding rate.

33. An apparatus that encodes video data, the apparatus comprising:

means to determine a size of a video queue in a radio link protocol (RLP) layer;

means to determine a transmit power headroom limitation from a medium access control (MAC) layer;

means to use the determined transmit power headroom limitation to determine a MAC payload size;

means to use the determined MAC payload size and an estimate of how many transmission opportunities are given to video in a time period to determine an estimated video throughput;

means to use the estimated video throughput and the determined size of the video queue in the RLP layer to determine a channel-constrained video frame size; and

means to use the channel-constrained video frame size to control a video encoding rate.