Synchronization of audio and video data in a wireless communication system

• US 9,717,018 B2
• Filed: 05/13/2005
• Issued: 07/25/2017
• Est. Priority Date: 05/13/2004
• Status: Expired due to Fees

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Garudadri, Harinath, Sagetong, Phoom, Nanda, Sanjiv
• Primary Examiner(s): Phung, Luat
• Application Number:
  US11/129,635
• Publication Number:
  US 20050259694A1
• Time in Patent Office: 4,456 Days
• Field of Search: 370503, 370506, 370230, 370412, 370468, 370476, 370509, 709247, 725 67, 725 68, 725 81, 379 671, 382245
• CPC Class Codes: H04L 47/36 by determining packet size, e.g. maximum transfer unit [MTU]
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/75 Media network packet handling
• CPC Class Codes: H04L 65/764 at the destination reformatting of additional data in video clients H04N21/4355
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04L 69/04 Protocols for data compression, e.g. ROHC
• CPC Class Codes: H04L 69/161 Implementation details of TCP/IP or UDP/IP stack architecture; Specification of modified or new header fields
• CPC Class Codes: H04L 69/166 IP fragmentation; TCP segmentation
• CPC Class Codes: H04L 69/22 Parsing or analysis of headers
• CPC Class Codes: H04L 69/321 Interlayer communication protocols or service data unit [SDU] definitions; Interfaces between layers
• CPC Class Codes: H04L 9/40 Network security protocols
• CPC Class Codes: H04N 19/102 characterised by the element, parameter or selection affected or controlled by the adaptive coding
• CPC Class Codes: H04N 19/115 Selection of the code volume for a coding unit prior to coding
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/152 by measuring the fullness of the transmission buffer
• CPC Class Codes: H04N 19/164 Feedback from the receiver or from the transmission channel
• CPC Class Codes: H04N 19/174 the region being a slice, e.g. a line of blocks or a group of blocks
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 21/2381 Adapting the multiplex stream to a specific network, e.g. an Internet Protocol [IP] network transmission of MPEG streams over ATM H04L12/5601
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/44004 : involving video buffer management, e.g. video decoder buffer or video display buffer
• CPC Class Codes: H04N 21/4788 : communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 21/6131 : involving transmission via a mobile phone network wireless downlink channel access H04W74/006
• CPC Class Codes: H04N 21/6181 : involving transmission via a mobile phone network arrangements for providing broadcast or conference services to substation in data switching networks in combination with wireless systems H04L12/189; wireless uplink channel access H04W74/004
• CPC Class Codes: H04N 21/6437 : Real-time Transport Protocol [RTP]
• CPC Class Codes: H04N 21/64707 : for transferring content from a first network to a second network, e.g. between IP and wireless
• CPC Class Codes: H04W 28/06 : Optimizing the usage of the radio link, e.g. header compression, information sizing , discarding information system modifying transmission characteristic according to link quality by modifying frame length H04L1/0007; dynamic adaptation of the packet size for flow control or congestion control H04L47/365
• CPC Class Codes: H04W 28/065 : using assembly or disassembly of packets
• CPC Class Codes: H04W 4/06 : Selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS]; Services to user groups; One-way selective calling services
• CPC Class Codes: H04W 72/044 : based on the type of the allocated resource
• CPC Class Codes: H04W 72/1263 : Mapping of traffic onto schedule, e.g. scheduled allocation or multiplexing of flows
• CPC Class Codes: H04W 80/00 : Wireless network protocols or protocol adaptations to wireless operation
• CPC Class Codes: H04W 84/04 : Large scale networks; Deep hierarchical networks
• CPC Class Codes: H04W 88/02 : Terminal devices
• CPC Class Codes: H04W 88/181 : Transcoding devices; Rate adaptation devices

Abstract

Techniques are described for encoding an audio video stream that is transmitted over a network, for example a wireless or IP network, such that an entire frame of audio and an entire frame of video are transmitted simultaneously within a period required to render the audio video stream frames by an application in a receiver. Aspects of the techniques include receiving audio and video RTP streams and assigning an entire frame of RTP video data to communication channel packets that occupy the same period, or less, as the video frame rate. Also an entire frame of RTP audio data is assigned to communication channel packets that occupy the same period, or less, as the audio frame rate. The video and audio communication channel packets are transmitted simultaneously. Receiving and assigning RTP streams can be performed in a remote station, or a base station.

Claims

1. A data stream synchronizer, comprising:

a communication channel interface configured to receive a plurality of communication channel packets over a variable capacity communication channel via a wireless communication network, wherein the plurality of communication channel packets includes;

first blocks of communication channel packets, where each block in the first blocks of communication channel packets corresponds to a respective video frame that is encoded into an encoded video data stream, wherein the encoded video data stream is encoded from video frames of varying sizes, and wherein each block in the first blocks of communication channel packets occupies a period that is the same or less than a video frame period of the encoded video data stream irrespective of a size of the block based on a channel capacity of the variable capacity communication channel being dynamically varied to accommodate the size of the block; and

second blocks of communication channel packets, where each block in the second blocks of communication channel packets corresponds to a respective audio frame that is encoded into an encoded audio data stream;

a first decoder coupled to the communication channel interface to receive the first blocks of communication channel packets corresponding to the encoded video data stream and to output a decoded video data stream;

a second decoder coupled to the communication channel interface to receive the second blocks of communication channel packets corresponding to the encoded audio data stream and to output a decoded audio data stream;

a first buffer configured to accumulate the decoded video data stream and to output one frame of the decoded video data stream each video frame period;

a second buffer configured to accumulate the decoded audio data stream and to output one frame of the decoded audio data stream each audio frame period; and

a combiner configured to receive the one frame of the decoded video data stream and the one frame of the decoded audio data stream and to output a synchronized frame of audio/video data every video frame period, wherein the output synchronized frame of audio/video data includes only one frame of audio data per video frame period.

2. The data stream synchronizer of claim 1, wherein each block in the second blocks of communication channel packets occupies a period that is the same or less than the video frame period of the encoded video data stream.

3. The data stream synchronizer of claim 1, wherein each block in the first blocks of communication channel packets occupies a period that is the same occupied by each block in the second blocks of communication channel packets.

4. The data stream synchronizer of claim 1, wherein each block in the first blocks of communication channel packets includes the same number of communication channel packets as each of the other blocks in the first blocks of communication channel packets.

5. The data stream synchronizer of claim 4, wherein each block in the first block of communication channel packets has a variable capacity adjustable based on the video frames of varying sizes.

6. The data stream synchronizer of claim 1, wherein each block in the first blocks of communication channel packets includes the same number of communication channel packets as included in each block in the second blocks of communication channel packets.

7. The data stream synchronizer of claim 1, wherein at least one block in the first blocks of communication channel packets includes a blank communication channel packet.

8. A remote station apparatus, comprising:

a communication channel interface configured to receive a plurality of communication channel packets over a variable capacity communication channel via a wireless communication network, wherein the plurality of communication channel packets includes;

first blocks of communication channel packets, where each block in the first blocks of communication channel packets corresponds to a respective video frame that is encoded into an encoded video data, wherein the encoded video data is encoded from video frames of varying sizes, and wherein each block in the first blocks of communication channel packets occupies a period that is the same or less than a video frame period of the encoded video data irrespective of a size of the block based on a channel capacity of the variable capacity communication channel being dynamically varied to accommodate the size of the block; and

second blocks of communication channel packets, where each block in the second blocks of communication channel packets corresponds to a respective audio frame that is encoded into an encoded audio data;

a video decoder coupled to the communication channel interface to receive the first blocks of communication channel packets corresponding to the encoded video data and to output decoded video data;

an audio decoder coupled to the communication channel interface to receive the second blocks of communication channel packets corresponding to the encoded audio data and to output decoded audio data;

a video buffer configured to accumulate the decoded video data for at least one video frame period and to output one frame of the decoded video data each video frame period;

an audio buffer configured to accumulate the decoded audio data for multiple audio frame periods and to output one frame of the decoded audio data each audio frame period; and

a combiner configured to receive the one frame of the decoded video data and the one frame of the decoded audio data and configured to output a synchronized frame of decoded audio/video data every video frame period, wherein the output synchronized frame of decoded audio/video data includes only one frame of audio data per video frame period.

9. The remote station apparatus of claim 8, wherein the video decoder is at least one of an MPEG decoder, H.263 decoder, and H.264 decoder.

10. The remote station apparatus of claim 8, wherein the audio decoder is at least one of an MPEG decoder, H.263 decoder, and H.264 decoder.

11. The remote station apparatus of claim 8, further comprising a control processor configured to control the audio decoder, the video decoder, and the combiner.

12. A base station apparatus, comprising:

a communication channel interface configured to receive a plurality of communication channel packets over a variable capacity communication channel via a wireless communication network, wherein the plurality of communication channel packets includes;

first blocks of communication channel packets, where each block in the first blocks of communication channel packets corresponds to a respective video frame that is encoded into an encoded video data, wherein the encoded video data is encoded from video frames of varying sizes, and wherein each block in the first blocks of communication channel packets occupies a period that is the same or less than a video frame period of the encoded video data irrespective of a size of the block based on a channel capacity of the variable capacity communication channel being dynamically varied to accommodate the size of the block; and

second blocks of communication channel packets, where each block in the second blocks of communication channel packets corresponds to a respective audio frame that is encoded into an encoded audio data;

a video decoder coupled to the communication channel interface to receive the first blocks of communication channel packets corresponding to the encoded video data and to output decoded video data;

an audio decoder coupled to the communication channel interface to receive the second blocks of communication channel packets corresponding to the encoded audio data via the wireless communication network and to output decoded audio data;

a video buffer configured to accumulate the decoded video data for a video frame period and to output one frame of the decoded video data during the video frame period;

an audio buffer configured to accumulate the decoded audio data for an audio frame period and to output one frame of the decoded audio data during the audio frame period; and

a combiner configured to receive the one frame of the decoded video data and the one frame of the decoded audio data and to output a synchronized frame of audio/video data every video frame period, wherein the output synchronized frame of audio/video data includes only one frame of audio data per video frame period.

13. The base station apparatus of claim 12, wherein the video decoder is at least one of an MPEG decoder, H.263 decoder, and H.264 decoder.

14. The base station apparatus of claim 12, wherein the audio decoder is at least one of an MPEG decoder, H.263 decoder, and H.264 decoder.

15. The base station apparatus of claim 12, further comprising a control processor configured to control the audio decoder, the video decoder, and the combiner.

16. A method for decoding and synchronizing data streams, comprising:

receiving a plurality of communication channel packets over a variable capacity communication channel via a wireless communication network, wherein the plurality of communication channel packets includes;

first blocks of communication channel packets, where each block in the first blocks of communication channel packets corresponds to a respective video frame that is encoded into an encoded video data stream, wherein the encoded video data stream is encoded from video frames of varying sizes, and wherein each block in the first blocks of communication channel packets occupies a period that is the same or less than a video frame period of the encoded video data stream irrespective of a size of the block based on a channel capacity of the variable capacity communication channel being dynamically varied to accommodate the size of the block; and

second blocks of communication channel packets, where each block in the second blocks of communication channel packets corresponds to a respective audio frame that is encoded into an encoded audio data stream;

decoding the encoded video data stream, and outputting a decoded video data stream;

decoding the encoded audio data stream, and outputting a decoded audio data stream;

accumulating the decoded video data stream and outputting one frame of the decoded video data stream each video frame period;

accumulating the decoded audio data stream and outputting one frame of the decoded audio data stream each audio frame period; and

combining the one frame of the decoded video data stream with the one frame of the decoded audio data stream and outputting a synchronized frame of audio/video data every video frame period, wherein the output synchronized frame of audio/video data includes only one frame of audio data per video frame period.

17. A method for decoding and synchronizing audio and video data, comprising:

receiving a plurality of communication channel packets over a variable capacity communication channel via a wireless communication network, wherein the plurality of communication channel packets includes;

first blocks of communication channel packets, where each block in the first blocks of communication channel packets corresponds to a respective video frame that is encoded into an encoded video data, wherein the encoded video data is encoded from video frames of varying sizes, and wherein each block in the first blocks of communication channel packets occupies a period that is the same or less than a video frame period of the encoded video data irrespective of a size of the block based on a channel capacity of the variable capacity communication channel being dynamically varied to accommodate the size of the block; and

second blocks of communication channel packets, where each block in the second blocks of communication channel packets corresponds to a respective audio frame that is encoded into an encoded audio data;

outputting decoded video data in response to the encoded video data;

outputting decoded audio data in response to the encoded audio data;

accumulating the decoded video data for a video frame period and outputting one frame of the decoded video data each video frame period;

accumulating the decoded audio data for an audio frame period and outputting one frame of the decoded audio data each audio frame period; and

combining the one frame of the decoded video data with the one frame of the decoded audio data and outputting a synchronized frame of decoded audio/video data every video frame period, wherein the output synchronized frame of decoded audio/video data includes only one frame of audio data per video frame period.

18. A non-transitory computer-readable media, comprising instructions stored thereon that, if executed by a processor, cause the processor to control execution of a method for decoding and synchronizing data streams, the method comprising:

receiving a plurality of communication channel packets over a variable capacity communication channel via a wireless communication network, wherein the plurality of communication channel packets includes;

first blocks of communication channel packets, where each block in the first blocks of communication channel packets corresponds to a respective video frame that is encoded into an encoded video data stream, wherein the encoded video data stream includes is encoded from video frames of varying sizes, and wherein each block in the first blocks of communication channel packets occupies a period that is the same or less than a video frame period of the encoded video data stream irrespective of a size of the block based on a channel capacity of the variable capacity communication channel being dynamically varied to accommodate the size of the block; and

second blocks of communication channel packets, where each block in the second blocks of communication channel packets corresponds to a respective audio frame that is encoded into an encoded audio data stream;

decoding the encoded video data stream, and outputting a decoded video data stream;

decoding the encoded audio data stream, and outputting a decoded audio data stream;

accumulating the decoded video data stream and outputting one frame of the decoded video data stream each video frame period;

accumulating the decoded audio data stream and outputting one frame of the decoded audio data stream each audio frame period; and

combining the one frame of the decoded video data stream with the one frame of the decoded audio data stream and outputting a synchronized frame of audio/video data every video frame period, wherein the output synchronized frame of audio/video data includes only one frame of audio data per video frame period.

19. The non-transitory computer-readable media of claim 18, further comprising a remote station apparatus or a base station apparatus integrated with the non-transitory computer-readable media.

20. A non-transitory computer-readable media, comprising instructions stored thereon that, if executed by a processor, cause the processor to control execution of a method for decoding and synchronizing audio and video data, the method comprising:

receiving a plurality of communication channel packets over a variable capacity communication channel via a wireless communication network, wherein the plurality of communication channel packets includes;

first blocks of communication channel packets, where each block in the first blocks of communication channel packets corresponds to a respective video frame that is encoded into an encoded video data stream, wherein the encoded video data stream is encoded from video frames of varying sizes, and wherein each block in the first blocks of communication channel packets occupies a period that is the same or less than a video frame period of the encoded video data stream irrespective of a size of the block based on a channel capacity of the variable capacity communication channel being dynamically varied to accommodate the size of the block; and

second blocks of communication channel packets, where each block in the second blocks of communication channel packets corresponds to a respective audio frame that is encoded into an encoded audio data stream;

and outputting decoded video data in response to the encoded video data stream;

receiving encoded audio data via the wireless communication network and outputting decoded audio data;

accumulating the decoded video data for a video frame period and outputting one frame of the decoded video data each video frame period;

accumulating the decoded audio data for an audio frame period and outputting one frame of the decoded audio data each audio frame period; and

combining the one frame of the decoded video data with the one frame of the decoded audio data and outputting a synchronized frame of decoded audio/video data every video frame period, wherein the output synchronized frame of decoded audio/video data includes only one frame of audio data per video frame period.

21. The non-transitory computer-readable media of claim 20, further comprising a remote station apparatus or a base station apparatus integrated with the non-transitory computer-readable media.

22. A data stream synchronizer, comprising:

means for receiving a plurality of communication channel packets over a variable capacity communication channel via a wireless communication network, wherein the plurality of communication channel packets includes;

first blocks of communication channel packets, where each block in the first blocks of communication channel packets corresponds to a respective video frame that is encoded into an encoded video data stream, wherein the encoded video data stream is encoded from video frames of varying sizes, and wherein each block in the first blocks of communication channel packets occupies a period that is the same or less than a video frame period of the encoded video data stream irrespective of a size of the block based on a channel capacity of the variable capacity communication channel being dynamically varied to accommodate the size of the block; and

second blocks of communication channel packets, where each block in the second blocks of communication channel packets corresponds to a respective audio frame that is encoded into an encoded audio data stream;

means for decoding the encoded video data stream and to output a decoded video data stream;

means for decoding the encoded audio data stream and to output a decoded audio data stream;

means for accumulating the decoded video data stream and to output one frame of the decoded video data stream each video frame period;

means for accumulating the decoded audio data stream and to output one frame of the decoded audio data stream each audio frame period;

means for buffering the frames of the decoded audio and video data streams; and

means for combining the one frame of the decoded video data stream with the one frame of the decoded audio data stream and to output a synchronized frame of audio/video data every video frame period, wherein the output synchronized frame of audio/video data includes only one frame of audio data per video frame period.

23. A remote station apparatus, comprising:

means for receiving a plurality of communication channel packets over a variable capacity communication channel via a wireless communication network, wherein the plurality of communication channel packets includes;

first blocks of communication channel packets, where each block in the first blocks of communication channel packets corresponds to a respective video frame that is encoded into an encoded video data, wherein the encoded video data is encoded from video frames of varying sizes, and wherein each block in the first blocks of communication channel packets occupies a period that is the same or less than a video frame period of the encoded video data irrespective of a size of the block based on a channel capacity of the variable capacity communication channel being dynamically varied to accommodate the size of the block; and

second blocks of communication channel packets, where each block in the second blocks of communication channel packets corresponds to a respective audio frame that is encoded into an encoded audio data;

means for outputting decoded video data in response to the encoded video data;

means for outputting decoded audio data in response to the encoded audio data;

means for accumulating the decoded video data for a video frame period and outputting one frame of the decoded video data each video frame period;

means for accumulating the decoded audio data for an audio frame period and outputting one frame of the decoded audio data each audio frame period;

means for buffering the frames of the decoded audio and video data; and

means for combining the one frame of the decoded video data with the one frame of the decoded audio data and outputting a synchronized frame of audio/video data every video frame period, wherein the output synchronized frame of audio/video data includes only one frame of audio data per video frame period.

24. A base station apparatus, comprising:

means for receiving a plurality of communication channel packets over a variable capacity communication channel via a wireless communication network, wherein the plurality of communication channel packets includes;

first blocks of communication channel packets, where each block in the first blocks of communication channel packets corresponds to a respective video frame that is encoded into an encoded video data, wherein the encoded video data is encoded from video frames of varying sizes, and wherein each block in the first blocks of communication channel packets occupies a period that is the same or less than a video frame period of the encoded video data irrespective of a size of the block based on a channel capacity of the variable capacity communication channel being dynamically varied to accommodate the size of the block; and

second blocks of communication channel packets, where each block in the second blocks of communication channel packets corresponds to a respective audio frame that is encoded into an encoded audio data;

means for outputting decoded video data in response to the encoded video data;

means for outputting decoded audio data in response to the encoded audio data;

means for accumulating the decoded video data for a video frame period and outputting one frame of the decoded video data each video frame period;

means for accumulating the decoded audio data for an audio frame period and outputting one frame of the decoded audio data each audio frame period;

means for buffering the decoded audio and video data; and

means for combining the one frame of the decoded video data with the one frame of the decoded audio data and outputting a synchronized frame of audio/video data every video frame period, wherein the output synchronized frame of audio/video data includes only one frame of audio data per video frame period.

25. A method for decoding and synchronizing data streams, comprising:

receiving a plurality of communication channel packets over a variable capacity communication channel via a wireless communication network, wherein the plurality of communication channel packets includes;

first blocks of communication channel packets, where each block in the first blocks of communication channel packets corresponds to a respective video frame that is encoded into an encoded video data stream, wherein the encoded video data stream is encoded from video frames of varying sizes, and wherein each block in the first blocks of communication channel packets occupies a period that is the same or less than a video frame period of the encoded video data stream irrespective of a size of the block based on a channel capacity of the variable capacity communication channel being dynamically varied to accommodate the size of the block; and

second blocks of communication channel packets, where each block in the second blocks of communication channel packets corresponds to a respective audio frame that is encoded into an encoded audio data stream;

decoding the encoded video data stream into a decoded video data stream;

decoding an encoded audio data stream received via the wireless communication network into a decoded audio data stream;

accumulating the decoded video data stream and outputting one frame of the decoded video data stream each video frame period;

accumulating the decoded audio data stream and outputting one frame of the decoded audio data stream each audio frame period; and

combining the one frame of the decoded video data stream with the one frame of the decoded audio data stream and outputting a synchronized frame of audio/video data every video frame period, wherein the output synchronized frame of audio/video data includes only one frame of audio data per video frame period.

26. A non-transitory computer-readable media, comprising instructions stored thereon that, if executed by a processor, cause the processor to control execution of a method for decoding and synchronizing data streams, the method comprising:

receiving a plurality of communication channel packets over a variable capacity communication channel via a wireless communication network, wherein the plurality of communication channel packets includes;

first blocks of communication channel packets, where each block in the first blocks of communication channel packets corresponds to a respective video frame that is encoded into an encoded video data stream, wherein the encoded video data stream is encoded from video frames of varying sizes, and wherein each block in the first blocks of communication channel packets occupies a period that is the same or less than a video frame period of the encoded video data stream irrespective of a size of the block based on a channel capacity of the variable capacity communication channel being dynamically varied to accommodate the size of the block; and

second blocks of communication channel packets, where each block in the second blocks of communication channel packets corresponds to a respective audio frame that is encoded into an encoded audio data stream;

decoding the encoded video data stream into a decoded video data stream;

decoding the encoded audio data stream into a decoded audio data stream;

accumulating the decoded video data stream and outputting one frame of the decoded video data stream each video frame period;

accumulating the decoded audio data stream and outputting one frame of the decoded audio data stream each audio frame period; and

combining the one frame of the decoded video data stream with the one frame of the decoded audio data stream and outputting a synchronized frame of audio/video data every video frame period, wherein the output synchronized frame of audio/video data includes only one frame of audio data per video frame period.

27. A data stream synchronizer, comprising:

means for receiving a plurality of communication channel packets over a variable capacity communication channel via a wireless communication network, wherein the plurality of communication channel packets includes;

first blocks of communication channel packets, where each block in the first blocks of communication channel packets corresponds to a respective video frame that is encoded into an encoded video data stream, wherein the encoded video data stream is encoded from video frames of varying sizes, and wherein each block in the first blocks of communication channel packets occupies a period that is the same or less than a video frame period of the encoded video data stream irrespective of a size of the block based on a channel capacity of the variable capacity communication channel being dynamically varied to accommodate the size of the block; and

second blocks of communication channel packets, where each block in the second blocks of communication channel packets corresponds to a respective audio frame that is encoded into an encoded audio data stream;

means for decoding the encoded video data stream into a decoded video data stream;

means for decoding the encoded audio data stream into a decoded audio data stream;

means for accumulating the decoded video data stream and outputting one frame of the decoded video data stream each video frame period;

means for accumulating the decoded audio data stream and outputting one frame of the decoded audio data stream each audio frame period;

means for buffering the one frame of the decoded video data stream and the one frame of the decoded audio data stream, wherein the buffering means are sized at least partially based on a maximum delay experienced during transmission of the encoded video and audio data streams; and

means for combining the one frame of the decoded video data stream with the one frame of the decoded audio data stream and for outputting a synchronized frame of audio/video data every video frame period, wherein the output synchronized frame of audio/video data includes only one frame of audio data per video frame period.