Efficient video packetization methods for packet-switched video telephony applications

• US 8,379,733 B2
• Filed: 09/26/2006
• Issued: 02/19/2013
• Est. Priority Date: 09/26/2006
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Lee, Yen-Chi, Olson, Jill
• Primary Examiner(s): Perungavoor, Sath V
• Assistant Examiner(s): Pe, Geepy
• Application Number:
  US11/535,380
• Publication Number:
  US 20080084933A1
• Time in Patent Office: 2,338 Days
• Field of Search: 375/240.01, 375/240.02, 375/240.08, 375/240.12, 375/240.26
• US Class Current: 375/240.26
• CPC Class Codes: H04N 19/68 involving the insertion of resynchronisation markers into the bitstream
• CPC Class Codes: H04N 19/89 involving methods or arrangements for detection of transmission errors at the decoder
• CPC Class Codes: H04N 21/236 Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream multiplexing of data packets for data networks, e.g. RTP/UDP H04L65/00
• CPC Class Codes: H04N 21/2381 Adapting the multiplex stream to a specific network, e.g. an Internet Protocol [IP] network transmission of MPEG streams over ATM H04L12/5601
• CPC Class Codes: H04N 21/2389 Multiplex stream processing, e.g. multiplex stream encrypting
• CPC Class Codes: H04N 21/43637 involving a wireless protocol, e.g. Bluetooth, RF or wireless LAN [IEEE 802.11] arrangements for wireless networking or broadcasting of information in indoor or near-field type systems H04B10/114
• CPC Class Codes: H04N 21/6131 involving transmission via a mobile phone network wireless downlink channel access H04W74/006
• CPC Class Codes: H04N 21/64322 IP
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q

Abstract

A device having an encoder and a packet switched video assembler which first estimates how many packets should be generated for one frame based on an estimated frame size and an initial fixed target packet size. Then, an updated packet size is recalculated on a packet-by-packet or slice-by-slice basis. The video assembler includes a three-stage process. The first stage estimates the frame size with or without encoding the video sequence, wherein in the case where the first stage includes encoding, the encoded data is packed in to a simulation bitstream. The second stage estimates the number of packets using the target packet size, estimated frame size and a configuration mode. The third stage dynamically updates the packet size using one-MB look-ahead approach with pre-encoding each MB. In an alternate embodiment, the third stage turns off the look ahead feature when encoding a current macroblock takes place during the third stage.

Claims

1. A packet-switched video telephony device comprising:

an encoder operable to encode a video sequence to create an encoded video data;

a packet switched video assembler operable to packetize the encoded video data into dynamically adjusted packets by estimating a size of a frame of the video sequence to determine an estimated frame size, estimating a number of packets to be generated for the frame based on the estimated frame size and iteratively calculating an updated packet size, based on at least the estimated frame size and the number of packets, on a packet-by-packet basis, while the encoded video data is packed into an actual bitstream; and

a transmitter for transmitting the actual bitstream.

2. The device of claim 1, wherein the packet switched video assembler is operable to determine the number of packets based on a configuration mode, the estimated frame size and an initial fixed target packet size, and calculate a new target packet size based on the number of packets and the estimated frame size.

3. The device of claim 2, wherein the packet switched video assembler is operable to estimate the estimated frame size by packing the encoded data into a simulation bitstream.

4. The device of claim 3, wherein the packet switched video assembler is operable to estimate the size of the frame to determine the estimated frame size by inserting a simulation resynchronization marker or simulation slice header in the simulation bitstream.

5. The device of claim 4, wherein the packet switched video assembler is operable to pack a current macroblock of the encoded data into the actual bitstream, look ahead one macroblock in the encoded video data, evaluate whether a result of a count representative of a size of the current packet size plus a predetermined portion of a count representative of a size of the one macroblock exceeds the updated packet size and, when the result exceeds the updated packet size and a packet count is less than the number of packets, insert an actual resynchronization marker or an actual slice header before the one macroblock and recalculate the updated packet size.

6. The device of claim 2, wherein the packet switched video assembler is operable to estimate the size of the frame to determine the estimated frame size by calculating a number of non-zero coefficients (NZCs) for a current macroblock of the encoded video data and accumulating the number of NZCs for each macroblock in the frame to form a total frame NZC.

7. The device of claim 6, wherein the packet switched video assembler is operable to estimate the size of the frame to determine the estimated frame size based on the following:

estimated frame size=alpha*total frame NZC+previous non-texture size

where alpha is defined by

alpha=(previous frame size−

previous non-texture size)/previous total frame NZC and

where the previous non-texture size is a count representative of non-texture bits in a previous frame, the previous frame size is an actual frame size of the previous frame and the previous total frame NZC is the total frame NZC of the previous frame.

8. The device of claim 7, wherein the packet switched video assembler is operable to, after packing of a current macroblock of the encoded data into the actual bitstream, look ahead one macroblock, evaluate whether a result of a count representative of a current packet size plus a count of the NZCs in the one macroblock multiplied by a predetermined factor is greater than the updated packet size and, when the result exceeds the updated packet size and a packet count is less than the number of packets, insert an actual resynchronization marker or an actual slice header before the one macroblock and recalculate the updated packet size.

9. The device of claim 1, wherein the packet switched video assembler is operable to estimate the size of the frame to determine the estimated frame size based on the following:

estimated_frame_size=alpha*desired total frame NZC+previous non-texture size

where alpha=previous frame size−

previous non-texture size)/previous total frame NZC and

where the previous non-texture size is a count representative of non-texture bits in a previous frame, the previous frame size is an actual frame size of the previous frame, the previous total frame NZC is a count representative of an accumulation of a previous frame'"'"'s total non-zero coefficients (NZC) and the desired total frame NZC is an initial fixed value identifying a frame'"'"'s total non-zero coefficients (NZC).

10. The device of claim 9, wherein the packet switched video assembler is operable to calculate a total frame NZC value representative of a frame'"'"'s total number of non-zero coefficients (NZC) for a maximum quantization parameter and calculate a target packet NZC value representative of a target packet'"'"'s number of non-zero coefficients (NZC) based on a rounded result of the total frame NZC value divided by the number of packets, the number of packets being a function of a configuration mode.

11. The device of claim 10, wherein the packet switched video assembler is operable to, after encoding a current macroblock, calculate a number of non-zero coefficients (NZC) for the current macroblock and packing the current macroblock into the actual bitstream, evaluate whether a count representative of a current packet'"'"'s non-zero coefficients (NZC) is greater than an updated packet NZC value and, when the result exceeds the updated packet NZC value and a packet count is less than the number of packets, insert a resynchronization marker or a slice header before a next macroblock and recalculate the updated packet NZC value.

12. A non-transitory computer-readable medium comprising program instructions that, when executed, cause one or more processors to:

encode a video sequence to create an encoded video data;

estimate a size of a frame of the video sequence to determine an estimated frame size, estimate a number of packets to be generated for a frame based on the estimated frame size; and

iteratively calculate an updated packet size, based on at least the estimated frame size and the number of packets, on a packet-by-packet basis, while the encoded video data is packed into an actual bitstream.

13. The non-transitory computer readable medium of claim 12, wherein the program instructions further cause the one or more processors to:

determine a configuration mode to vary a calculation for the number of packets; and

calculate a new target packet size based on the number of packets for the configuration mode and the estimated frame size.

14. The non-transitory computer readable medium of claim 13, wherein the program instructions further cause the one or more processors to estimate the size of the frame to determine the estimated frame size by packing the encoded data into a simulation bitstream.

15. The non-transitory computer readable medium of claim 14, wherein the program instructions further cause the one or more processors to insert a simulation resynchronization marker or simulation slice header in the simulation bitstream.

16. The non-transitory computer readable medium of claim 15, wherein the program instructions further cause the one or more processors to:

pack a current macroblock of the encoded data into the actual bitstream;

look ahead one macroblock in the encoded video data;

evaluate whether a result of a count representative of a size of the current packet size plus a predetermined portion of a count representative of a size of the one macroblock exceeds the updated packet size; and

when the result exceeds the updated packet size and a packet count is less than the number of packets, insert an actual resynchronization marker or an actual slice header before the one macroblock and recalculate the updated packet size.

17. The non-transitory computer readable medium of claim 13, wherein the program instructions further cause the one or more processors to estimate the size of the frame to determine the estimated frame size by calculating a number of non-zero coefficients (NZCs) for a current macroblock of the encoded video data and accumulating the number of NZCs for each macroblock in the frame to form a total frame NZC.

18. The non-transitory computer readable medium of claim 17, wherein the program instructions further cause the one or more processors to calculate the size of the frame to determine the estimated frame size based on the following:

estimated frame size=alpha*total frame NZC+previous non-texture size

where alpha is defined by

alpha =(previous frame size−

previous non-texture size)/previous total frame NZC and

where the previous non-texture size is a count representative of non-texture bits in a previous frame, the previous frame size is an actual frame size of the previous frame and the previous total frame NZC is the total frame NZC of the previous frame.

19. The non-transitory computer readable medium of claim 18, wherein the program instructions further cause the one or more processors to:

after packing a current macroblock of the encoded data into the actual bitstream to look ahead one macroblock, evaluate whether a result of a count representative of a current packet size plus a count of the NZCs in the one macroblock multiplied by a predetermined factor is greater than the updated packet size; and

when the result exceeds the updated packet size and a packet count is less than the number of packets, insert an actual resynchronization marker or an actual slice header before the one macroblock and recalculate the updated packet size.

20. The non-transitory computer readable medium of claim 12, wherein the program instructions further cause the one or more processors to estimate the size of the frame to determine the estimated frame size based on the following:

estimated_frame_size=alpha*desired total frame NZC+previous non-texture size

where alpha=previous frame size−

previous non-texture size)/previous total frame NZC and

where the previous non-texture size is a count representative of non-texture bits in a previous frame, the previous frame size is an actual frame size of the previous frame, the previous total frame NZC is a count representative of an accumulation of a previous frame'"'"'s total non-zero coefficients (NZC) and the desired total frame NZC is an initial fixed value identifying a frame'"'"'s total non-zero coefficients (NZC).

21. The non-transitory computer readable medium of claim 20, wherein the program instructions further cause the one or more processors to calculate a total frame NZC value representative of a frame'"'"'s total number of non-zero coefficients (NZC) for a maximum quantization parameter and calculate a target packet NZC value representative of a target packet'"'"'s number of non-zero coefficients (NZC) based on a rounded result of the total frame NZC value divided by the number of packets, the number of packets being a function of a configuration mode.

22. The non-transitory computer readable medium of claim 21, wherein the program instructions further cause the one or more processors to:

after encoding a current macroblock, calculate a number of non-zero coefficients (NZC) for the current macroblock and packing the current macroblock into the actuale bitstream;

evaluate whether a count representative of a current packet'"'"'s non-zero coefficients (NZC) is greater than an updated packet NZC value; and

when the result exceeds the updated packet NZC value and a packet count is less than the number of packets, insert a resynchronization marker or a slice header before a next macroblock and recalculate the updated packet NZC value.

23. The non-transitory computer readable medium of claim 21, wherein the maximum quantization parameter is 31 for the encoded video data in a MPEG4 video coding standard format or a H263 video coding standard format, and 52 for the encoded video data in a H264 video coding standard format.

24. A packet-switched video telephony device comprising:

encoding means for encoding a video sequence to create an encoded video data;

video assembling means for assembling the encoded video data into dynamically adjusted packets by estimating a size of a frame of the video sequence to determine an estimated frame size, estimating a number of packets to be generated for a frame based on the estimated frame size and iteratively calculating an updated packet size, based on at least the estimated frame size and the number of packets, on a packet-by-packet basis, while the encoded video data is packed into an actual bitstream; and

transmitting means for transmitting the actual bitstream.

25. The device of claim 24, wherein the video assembling means includes:

determining means for determining the number of packets based on a configuration mode, the estimated frame size and an initial fixed target packet size; and

calculating means for calculating a new target packet size based on the number of packets and the estimated frame size.

26. The device of claim 25, wherein the video assembling means includes estimating means for estimating the size of the frame to determine the estimated frame size by packing the encoded data into a simulation bitstream.

27. The device of claim 26, wherein the estimating means includes inserting means for inserting a simulation resynchronization marker or simulation slice header in the simulation bitstream.

28. The device of claim 27, wherein the video assembling means includes:

packing means for packing a current macroblock of the encoded data into the actual bitstream;

looking ahead means for looking ahead one macroblock in the encoded video data;

evaluating means for evaluating whether a result of a count representative of a size of the current packet size plus a predetermined portion of a count representative of a size of the one macroblock exceeds the updated packet size;

inserting means for inserting an actual resynchronization marker or an actual slice header before the one macroblock, when the result exceeds the updated packet size; and

recalculating means for recalculating the updated packet size, when the result exceeds the updated packet size and a packet count is less than the number of packets.

29. The device of claim 25, wherein the video assembling means comprises:

estimating means for estimating the size of the frame to determine the estimated frame size by calculating a number of non-zero coefficients (NZCs) for a current macroblock of the encoded video data and accumulating the number of NZCs for each macroblock in the frame to form a total frame NZC.

30. The device of claim 29, wherein the estimating means for estimating the size of the frame to determine the estimated frame size calculates the estimated frame size based on the following:

estimated frame size=alpha*total frame NZC+previous non-texture size

where alpha is defined by

alpha=(previous frame size-previous−

non-texture size)/previous total frame NZC and

where the previous non-texture size is a count representative of non-texture bits in a previous frame, the previous frame size is an actual frame size of the previous frame and the previous total frame NZC is the total frame NZC of the previous frame.

31. The device of claim 30, wherein the video assembling means comprises:

looking ahead means for looking ahead one macroblock after packing a current macroblock of the encoded data into the actual bitstream;

evaluating means for evaluating whether a result of a count representative of a current packet size plus a count of the NZCs in the one macroblock multiplied by a predetermined factor is greater than the updated packet size;

inserting means for inserting an actual resynchronization marker or an actual slice header before the one macroblock when the result exceeds the updated packet size; and

recalculating means for recalculating the updated packet size, when the result exceeds the updated packet size and a packet count is less than the number of packets.

32. The device of claim 31, wherein the video assembling means comprise estimating means for estimating the size of the frame to determine the estimated frame size based on the following:

estimated_frame_size=alpha*desired total frame NZC+previous non-texture size

where alpha=previous frame size−

previous non-texture size)/previous total frame NZC and

where the previous non-texture size is a count representative of non-texture bits in a previous frame, the previous frame size is an actual frame size of the previous frame, the previous total frame NZC is a count representative of an accumulation of a previous frame'"'"'s total non-zero coefficients (NZC) and the desired total frame NZC is an initial fixed value identifying a frame'"'"'s total non-zero coefficients (NZC).

33. The device of claim 24, wherein the video assembling means comprises:

calculating means for calculating a total frame NZC value representative of a frame'"'"'s total number of non-zero coefficients (NZC) for a maximum quantization parameter; and

calculating means for calculating a target packet NZC value representative of a target packet'"'"'s number of non-zero coefficients (NZC) based on a rounded result of the total frame NZC value divided by the number of packets, the number of packets being a function of a configuration mode.

34. The device of claim 33, wherein the video assembling means comprises:

evaluating means for evaluating whether a count representative of a current packet'"'"'s non-zero coefficients (NZC) is greater than an updated packet NZC value, after encoding a current macroblock, calculating a number of non-zero coefficients (NZC) for the current macroblock and packing the current macroblock into the actual bitstream;

inserting means for inserting a resynchronization marker or a slice header before a next macroblock, when the result exceeds the updated packet NZC value; and

recalculating means for recalculating the updated packet NZC value, when the result exceeds the updated packet NZC value and a packet count is less than the number of packets.

35. The device of claim 34, wherein the maximum quantization parameter is 31 for the encoded video data in a MPEG4 video coding standard format or a H263 video coding standard format, and 52 for the encoded video data in a H264 video coding standard format.

36. The device of claim 24, wherein the device comprises one of a cellular telephone, wireless telephone, satellite telephone or other mobile/wireless communication device with a image capturing capability.

37. A method comprising:

encoding a video sequence to create encoded video data with an encoder; and

dynamically adjusting packets of the encoded video data with the encoder by estimating a size of a frame of the video sequence to determine an estimated frame size, estimating a number of packets to be generated for a frame, based on an estimated frame size, and iteratively calculating an updated packet size, based on at least the estimated frame size and the number of packets, on a packet-by-packet basis, while the encoded video data is packed into an actual bitstream.

38. The method of claim 37, wherein the dynamically adjusting step comprises the steps of:

determining the number of packets based on a configuration mode, the estimated frame size and an initial fixed target packet size; and

calculating a new target packet size based on the number of packets and the estimated frame size.

39. The method of claim 38, wherein estimating the size of the frame comprises estimating the size of the frame to determine the estimated frame size by packing the encoded data into a simulation bitstream.

40. The method of claim 39, wherein the estimating step for estimating the size of the frame to determine the estimated frame size includes the step of inserting a simulation resynchronization marker or simulation slice header in the simulation bitstream.

41. The method of claim 40, wherein dynamically adjusting step includes the steps of:

packing a current macroblock of the encoded data into the actual bitstream;

looking ahead one macroblock in the encoded video data;

evaluating whether a result of a count representative of a size of the current packet size plus a predetermined portion of a count representative of a size of the one macroblock exceeds the updated packet size; and

inserting an actual resynchronization marker or an actual slice header before the one macroblock, when the result exceeds the updated packet size and recalculating the updated packet size, when the result exceeds the updated packet size and a packet count is less than the number of packets.

42. The method of claim 38,

wherein estimating the size of the frame comprises estimating the size of the frame to determine the estimated frame size by calculating a number of non-zero coefficients (NZCs) for a current macroblock of the encoded video data

wherein the dynamically adjusting step comprises accumulating the number of NZCs for each macroblock in the frame to form a total frame NZC.

43. The method of claim 42, wherein the estimating step for estimating the size of the frame to determine the estimated frame size calculates the estimated frame size based on the following:

estimated frame size=alpha*total frame NZC +previous non-texture size

where alpha is defined by

alpha=(previous frame size−

previous non-texture size)/previous total frame NZC and

where the previous non-texture size is a count representative of non-texture bits in a previous frame, the previous frame size is an actual frame size of the previous frame and the previous total frame NZC is the total frame NZC of the previous frame.

44. The method of claim 43, wherein the dynamically adjusting step comprises the steps of:

after packing a current macroblock of the encoded data into the actual bitstream, looking ahead one macroblock;

evaluating whether a result of a count representative of a current packet size plus a count of the NZCs in the one macroblock multiplied by a predetermined factor is greater than the updated packet size;

inserting an actual resynchronization marker or an actual slice header before the one macroblock, when the result exceeds the updated packet size and a packet count is less than the number of packets; and

recalculating the updated packet size, when the result exceeds the updated packet size and a packet count is less than the number of packets.

45. The method of claim 37, wherein estimating the size of the frame to determine the estimated frame size comprises estimating the size of the frame based on the following:

estimated_frame_size=alpha*desired total frame NZC+previous non-texture size

where alpha=previous frame size−

previous non-texture size)/previous total frame NZC and

where the previous non-texture size is a count representative of non-texture bits in a previous frame, the previous frame size is actual frame size of the previous frame, the previous total frame NZC is a count representative of an accumulation of a previous frame'"'"'s total non-zero coefficients (NZC) and the desired total frame NZC is an initial fixed value identifying a frame'"'"'s total non-zero coefficients (NZC).

46. The method of claim 45, wherein the dynamically adjusting step comprises the steps of:

calculating a total frame NZC value representative of a frame'"'"'s total number of non-zero coefficients (NZC) for a maximum quantization parameter; and

calculating a target packet NZC value representative of a target packet'"'"'s number of non-zero coefficients (NZC) based on a rounded result of the total frame NZC value divided by the number of packets, the number of packets being a function of a configuration mode.

47. The method of claim 46, wherein the dynamically adjusting step comprises the steps of:

after encoding a current macroblock, calculating a number of non-zero coefficients (NZC) for the current macroblock and packing the current macroblock into the actual bitstream;

evaluating whether a count representative of a current packet'"'"'s non-zero coefficients (NZC) is greater than an updated packet NZC value;

inserting a resynchronization marker or a slice header before a next macroblock, when the result exceeds the updated packet NZC value and a packet count is less than the number of packets; and

recalculating the updated packet NZC value, when the result exceeds the updated packet NZC value and a packet count is less than the number of packets.

48. The method of claim 47, wherein the maximum quantization parameter is 31 for the encoded video data in a MPEG4 video coding standard format or a H263 video coding standard format, and 52 for the encoded video data in a H264 video coding standard format.

49. An encoder comprising:

a macroblock encoding module operable to encode a video sequence to create an encoded video data having macroblocks; and

a packet switched video assembler operable to packetize the encoded video data into dynamically adjusted packets by estimating a size of a frame of the video sequence to determine an estimated frame size, iteratively calculating an updated packet size, on a slice-by-slice basis, while the encoded video data is packed into an actual bitstream, the updated packet sized being a function of the estimated frame size derived from a fixed initial target frame size and an estimated number of packets.

50. The encoder of claim 49, wherein the estimated frame size is calculated by packing the encoded video data into a simulation bitstream and inserting a simulation resynchronization marker or simulation slice marker.