Video source rate control for video telephony

• US 20070097257A1
• Filed: 12/20/2005
• Published: 05/03/2007
• Est. Priority Date: 10/27/2005
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Lee, Yen-Chi, Tsai, Ming-Chang, El-Maleh, Khaled
• Granted Patent: US 8,548,048 B2
• Time in Patent Office: Days
• US Class Current: 348/419.1
• CPC Class Codes: H04N 19/115 Selection of the code volume for a coding unit prior to coding
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/147 according to rate distortion criteria rate-distortion as a criterion for motion estimation H04N19/567
• CPC Class Codes: H04N 19/152 by measuring the fullness of the transmission buffer
• CPC Class Codes: H04N 19/169 characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/184 the unit being bits, e.g. of the compressed video stream
• CPC Class Codes: H04N 19/188 the unit being a video data packet, e.g. a network abstraction layer [NAL] unit
• CPC Class Codes: H04N 19/194 involving only two passes
• CPC Class Codes: H04N 19/196 being specially adapted for the computation of encoding parameters, e.g. by averaging previously computed encoding parameters processing of motion vectors H04N19/513
• CPC Class Codes: H04N 19/198 including smoothing of a sequence of encoding parameters, e.g. by averaging, by choice of the maximum, minimum or median value
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 21/23406 involving management of server-side video buffer
• CPC Class Codes: H04N 21/2368 Multiplexing of audio and video streams
• CPC Class Codes: H04N 21/2381 Adapting the multiplex stream to a specific network, e.g. an Internet Protocol [IP] network transmission of MPEG streams over ATM H04L12/5601
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/4341 Demultiplexing of audio and video streams
• CPC Class Codes: H04N 21/44004 involving video buffer management, e.g. video decoder buffer or video display buffer
• CPC Class Codes: H04N 21/6125 : involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 21/6131 : involving transmission via a mobile phone network wireless downlink channel access H04W74/006
• CPC Class Codes: H04N 21/6175 : involving transmission via Internet broadcast-related systems characterised by the transmission system being the Internet H04H60/82
• CPC Class Codes: H04N 21/64322 : IP
• CPC Class Codes: H04N 21/6437 : Real-time Transport Protocol [RTP]

Abstract

The disclosure relates to techniques for video source rate control for video telephony (VT) applications. The source video encoding rate may controlled using a dual-buffer based estimation of a frame budget that defines a number of encoding bits available for a frame of the video. The dual-buffer based estimation technique may track the fullness of a physical video buffer and the fullness of the virtual video buffer. The source video encoding rate is then controlled based on the resulting frame budget. The contents of the virtual buffer depend on constraints imposed by a target encoding rate, while the contents of the physical buffer depend on constraints imposed by varying channel conditions. Consideration of physical video buffer fullness permits the video source rate control technique to be channel-adaptive. Consideration of virtual video buffer fullness permits the video source rate control technique to avoid encoding excessive video that could overwhelm the channel.

Claims

1. A method comprising:

adding encoded video to a physical video buffer;

removing a packet of the encoded video from the physical video buffer for transmission over a channel;

maintaining a virtual video buffer representing an amount of the encoded video relative to a target encoding rate; and

controlling an actual encoding rate of the video based on fullness of the physical video buffer and fullness of the virtual video buffer.

2. The method of claim 1, wherein the channel exhibits varying channel conditions, the method further removing varying amounts of the encoded video from the physical video buffer over time in response to the varying channel conditions.

3. The method of claim 1, wherein controlling an actual encoding rate includes skipping encoding of a video frame if a maximum of the fullness of the physical video buffer and the fullness of the virtual video buffer is greater than or equal to a skipping threshold.

4. The method of claim 3, further comprising estimating a frame budget defining a number of encoding bits available for a frame of the video if a maximum of the fullness of the physical video buffer and the fullness of the virtual video buffer is less than the skipping threshold, and controlling the actual encoding rate based on the frame budget.

5. The method of claim 4, wherein estimating a frame budget includes estimating the number of encoding bits based on a target bit rate, a target frame rate, the fullness of the physical video buffer, the fullness of the virtual video buffer, and the skipping threshold.

6. The method of claim 5, further comprising:

determining the fullness of the virtual video buffer substantially based on the following expression;

Wn=max(Wn-1+Bn-1−

R/F, 0)

where Wn is the fullness of the virtual video buffer for video frame n, Wn-1 is the fullness of the virtual video buffer for video frame n-1, Bn-1 is the actual coding bits allocated to video frame n-1, R is the target bit rate, and F is the target frame rate; and

determining the actual encoding rate substantially according to the following expressions;

Xn=max(w*Vn, Wn), and
Bn=R/F−

Δ

,

where Δ

=Xn/F if Xn>

Z*M, Δ

=Xn−

Z*M if Xn<

=Z*M, Vn is the fullness of the physical video buffer for video frame n, w is a positive integer, M is the skipping threshold, and Z is a coefficient.

7. The method of claim 1, further comprising estimating a frame budget defining a number of encoding bits available for a frame of the video if a maximum of the fullness of the physical video buffer and the fullness of the virtual video buffer is less than the skipping threshold, and controlling the actual encoding rate based on the frame budget.

8. The method of claim 7, wherein estimating a frame budget includes estimating the number of encoding bits based on a target bit rate, a target frame rate, the fullness of the physical video buffer, the fullness of the virtual video buffer, and the skipping threshold.

9. The method of claim 8, further comprising:

determining the fullness of the virtual video buffer substantially based on the following expression;

Wn=max(Wn-1+Bn-1−

R/F, 0)

where Wn is the fullness of the virtual video buffer for video frame n, Wn-1 is the fullness of the virtual video buffer for video frame n-1, Bn-1 is the actual coding bits allocated to video frame n-1, R is the target bit rate, and F is the target frame rate; and

determining the actual encoding rate substantially according to the following expressions;

Xn=max(w*Vn, Wn), and
Bn=R/F−

Δ

,

where Δ

=Xn/F if Xn>

Z*M, Δ

=Xn−

Z*M if Xn<

=Z*M, Vn is the fullness of the physical video buffer for video frame n, w is a positive integer, M is the skipping threshold, and Z is a coefficient.

10. The method of claim 1, wherein the channel is a wireless channel, the method further comprising:

estimating throughput of the wireless channel; and

generating a video packet from the physical video buffer with a video packet size determined based on the estimated throughput.

11. The method of claim 10, further comprising generating an audio packet, wherein the video packet size is determined based on the estimated throughput and a size of the audio packet.

12. The method of claim 11, further comprising generating a physical layer packet based on the audio packet and the video packet, and transmitting the physical layer packet to a remote device to support a mobile wireless video telephony conference.

13. The method of claim 12, wherein the physical layer packet conforms substantially to the cdma2000 1x EV-DO, Release 0, standard.

14. The method of claim 1, further comprising:

estimating a frame budget defining a number of encoding bits available for a frame of the video based on the actual encoding rate; and

allocating rho domain values to macroblocks within the frame based on the frame budget.

15. The method of claim 14, further comprising mapping the rho domain values to corresponding quantization parameter (QP) values to allocate a number of encoding bits to each of the macroblocks.

16. The method of claim 1, further comprising dropping at least a portion of the encoded video from the physical video buffer when the portion of the encoded video has been in the physical video buffer for a period of time that exceeds a threshold.

17. A system comprising:

a video encoder;

a physical video buffer that stores video encoded by the video encoder;

a packet shaper that removes the encoded video from the physical video buffer for transmission over a channel;

a virtual video buffer representing an amount of the encoded video relative to a target encoding rate; and

a controller that controls an actual encoding rate of the video based on fullness of the physical video buffer and fullness of the virtual video buffer.

18. The system of claim 17, wherein the channel exhibits varying channel conditions, and the packet shaper removes varying amounts of the encoded video from the physical video buffer over time in response to the varying channel conditions.

19. The system of claim 17, wherein the controller skips encoding of a video frame if a maximum of the fullness of the physical video buffer and the fullness of the virtual video buffer is greater than or equal to a skipping threshold.

20. The system of claim 19, wherein the controller estimates a frame budget defining a number of encoding bits available for a frame of the video if a maximum of the fullness of the physical video buffer and the fullness of the virtual video buffer is less than the skipping threshold, and controls the actual encoding rate based on the frame budget.

21. The system of claim 20, wherein the controller estimates the frame budget based on a target bit rate, a target frame rate, the fullness of the physical video buffer, the fullness of the virtual video buffer, and the skipping threshold.

22. The system of claim 21, wherein the controller:

determines the fullness of the virtual video buffer substantially based on the following expression;

Wn=max(Wn-1+Bn-1−

R/F, 0)

where Wn is the fullness of the virtual video buffer for video frame n, Wn-1 is the fullness of the virtual video buffer for video frame n-1, Bn-1 is the actual coding bits allocated to video frame n-1, R is the target bit rate, and F is the target frame rate; and

determines the actual encoding rate substantially according to the following expressions;

Xn=max(w*Vn, Wn), and
Bn=R/F−

Δ

,

where Δ

=Xn/F if Xn>

Z*M, Δ

=Xn−

Z*M if Xn<

=Z*M, Vn is the fullness of the physical video buffer for video frame n, w is a positive integer, M is the skipping threshold, and Z is a coefficient.

23. The system of claim 17, wherein the controller estimates a frame budget defining a number of encoding bits available for a frame of the video if a maximum of the fullness of the physical video buffer and the fullness of the virtual video buffer is less than the skipping threshold, and controls the actual encoding rate based on the frame budget.

24. The system of claim 23, wherein the controller estimates the frame budget based on a target bit rate, a target frame rate, the fullness of the physical video buffer, the fullness of the virtual video buffer, and the skipping threshold.

25. The system of claim 24, wherein the controller:

determines the fullness of the virtual video buffer substantially based on the following expression;

Wn=max(Wn-1+Bn-1−

R/F, 0)

where Wn is the fullness of the virtual video buffer for video frame n, Wn-1 is the fullness of the virtual video buffer for video frame n-1, Bn-1 is the actual coding bits allocated to video frame n-1, R is the target bit rate, and F is the target frame rate; and

determines the actual encoding rate according to the following expressions;

Xn=max(w*Vn, Wn), and
Bn=R/F−

Δ

,

where Δ

=Xn/F if Xn>

Z*M, Δ

=Xn−

Z*M if Xn<

=Z*M, Vn is the fullness of the physical video buffer for video frame n, w is a positive integer, M is the skipping threshold, and Z is a coefficient.

26. The system of claim 17, wherein the channel is a wireless channel, wherein the packet shaper estimates throughput of the wireless channel, and generates the video packet from the physical video buffer with a video packet size determined based on the estimated throughput.

27. The system of claim 26, further comprising an audio encoder that generates an audio packet, wherein the video packet size is determined based on the estimated throughput and a size of the audio packet.

28. The system of claim 27, further comprising a physical layer module that generates a physical layer packet based on the audio packet and the video packet, and transmits the physical layer packet to a remote device to support a mobile wireless video telephony conference.

29. The system of claim 28, wherein the physical layer packet conforms substantially to the cdma2000 1x EV-DO, Release 0, standard.

30. The system of claim 17, wherein the controller:

estimates a frame budget defining a number of encoding bits available for a frame of the video based on the actual encoding rate; and

allocates rho domain values to macroblocks within the frame based on the frame budget.

31. The system of claim 30, wherein the controller maps the rho domain values to corresponding quantization parameter (QP) values to allocate a number of encoding bits to each of the macroblocks.

32. The system of claim 17, wherein the physical video buffer drops at least a portion of the encoded video from the physical video buffer when the portion of the encoded video has been in the physical video buffer for a period of time that exceeds a threshold.

33. A system comprising:

means for encoding video;

physical video buffer means for storing the encoded video;

packet shaping means for generating a video packet from the encoded video from the physical video buffer means for transmission over a channel;

virtual buffer means for maintaining a virtual video buffer representing an amount of the encoded video relative to a target encoding rate; and

control means for controlling an actual encoding rate of the video based on fullness of the physical video buffer and fullness of the virtual video buffer.

34. The system of claim 33, wherein the channel exhibits varying channel conditions, and the packet shaping means removes varying amounts of the encoded video from the physical video buffer over time in response to the varying channel conditions.

35. The system of claim 33, wherein the control means includes means for skipping encoding of a video frame if a maximum of the fullness of the physical video buffer and the fullness of the virtual video buffer is greater than or equal to a skipping threshold.

36. The system of claim 35, wherein the control means includes means for estimating a frame budget defining a number of encoding bits available for a frame of the video if a maximum of the fullness of the physical video buffer and the fullness of the virtual video buffer is less than the skipping threshold, and means for controlling the actual encoding rate based on the frame budget.

37. The system of claim 36, wherein the means for estimating a frame budget includes means for estimating the number of encoding bits based on a target bit rate, a target frame rate, the fullness of the physical video buffer, the fullness of the virtual video buffer, and the skipping threshold.

38. The system of claim 33, wherein the channel is a wireless channel, and the packet shaping means includes means for estimating throughput of the wireless channel, and means for generating a video packet from the physical video buffer with a video packet size determined based on the estimated throughput.

39. The system of claim 38, further comprising means for generating an audio packet, wherein the packet shaping means determines the video packet size based on the estimated throughput and a size of the audio packet.

40. The system of claim 38, further comprising means for generating a physical layer packet based on the audio packet and the video packet, and means for transmitting the physical layer packet to a remote device to support a mobile wireless video telephony conference.

41. The system of claim 40, wherein the physical layer packet conforms substantially to the cdma2000 1x EV-DO, Release 0, standard.

42. The system of claim 33, wherein the control means includes:

means for estimating a frame budget defining a number of encoding bits available for a frame of the video based on the actual encoding rate;

means for allocating rho domain values to macroblocks within the frame based on the frame budget; and

means for mapping the rho domain values to corresponding quantization parameter (QP) values to allocate a number of encoding bits to each of the macroblocks.

43. The method of claim 33, further comprising means for dropping at least a portion of the encoded video from the physical video buffer means when the portion of the encoded video has been in the physical video buffer means for a period of time that exceeds a threshold.

44. A computer-readable medium comprising instructions to cause one or more processors to:

add encoded video to a physical video buffer;

remove a packet of the encoded video from the physical video buffer for transmission over a channel;

maintain a virtual video buffer representing an amount of the encoded video relative to a target encoding rate; and

control an actual encoding rate of the video based on fullness of the physical video buffer and fullness of the virtual video buffer.