Synchronization of audio and video data in a wireless communication system

• US 20050259694A1
• Filed: 05/13/2005
• Published: 11/24/2005
• Est. Priority Date: 05/13/2004
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Nanda, Sanjiv, Garudadri, Harinath, Sagetong, Phoom
• Granted Patent: US 9,717,018 B2
• Time in Patent Office: Days
• US Class Current: 370/503
• CPC Class Codes: H04L 47/36 by determining packet size, e.g. maximum transfer unit [MTU]
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/75 Media network packet handling
• CPC Class Codes: H04L 65/764 at the destination reformatting of additional data in video clients H04N21/4355
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04L 69/04 Protocols for data compression, e.g. ROHC
• CPC Class Codes: H04L 69/161 Implementation details of TCP/IP or UDP/IP stack architecture; Specification of modified or new header fields
• CPC Class Codes: H04L 69/166 IP fragmentation; TCP segmentation
• CPC Class Codes: H04L 69/22 Parsing or analysis of headers
• CPC Class Codes: H04L 69/321 Interlayer communication protocols or service data unit [SDU] definitions; Interfaces between layers
• CPC Class Codes: H04L 9/40 Network security protocols
• CPC Class Codes: H04N 19/102 characterised by the element, parameter or selection affected or controlled by the adaptive coding
• CPC Class Codes: H04N 19/115 Selection of the code volume for a coding unit prior to coding
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/152 by measuring the fullness of the transmission buffer
• CPC Class Codes: H04N 19/164 Feedback from the receiver or from the transmission channel
• CPC Class Codes: H04N 19/174 the region being a slice, e.g. a line of blocks or a group of blocks
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 21/2381 Adapting the multiplex stream to a specific network, e.g. an Internet Protocol [IP] network transmission of MPEG streams over ATM H04L12/5601
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/44004 : involving video buffer management, e.g. video decoder buffer or video display buffer
• CPC Class Codes: H04N 21/4788 : communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 21/6131 : involving transmission via a mobile phone network wireless downlink channel access H04W74/006
• CPC Class Codes: H04N 21/6181 : involving transmission via a mobile phone network arrangements for providing broadcast or conference services to substation in data switching networks in combination with wireless systems H04L12/189; wireless uplink channel access H04W74/004
• CPC Class Codes: H04N 21/6437 : Real-time Transport Protocol [RTP]
• CPC Class Codes: H04N 21/64707 : for transferring content from a first network to a second network, e.g. between IP and wireless
• CPC Class Codes: H04W 28/06 : Optimizing the usage of the radio link, e.g. header compression, information sizing , discarding information system modifying transmission characteristic according to link quality by modifying frame length H04L1/0007; dynamic adaptation of the packet size for flow control or congestion control H04L47/365
• CPC Class Codes: H04W 28/065 : using assembly or disassembly of packets
• CPC Class Codes: H04W 4/06 : Selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS]; Services to user groups; One-way selective calling services
• CPC Class Codes: H04W 72/044 : based on the type of the allocated resource
• CPC Class Codes: H04W 72/1263 : Mapping of traffic onto schedule, e.g. scheduled allocation or multiplexing of flows
• CPC Class Codes: H04W 80/00 : Wireless network protocols or protocol adaptations to wireless operation
• CPC Class Codes: H04W 84/04 : Large scale networks; Deep hierarchical networks
• CPC Class Codes: H04W 88/02 : Terminal devices
• CPC Class Codes: H04W 88/181 : Transcoding devices; Rate adaptation devices

Abstract

Techniques are described for encoding an audio video stream that is transmitted over a network, for example a wireless or IP network, such that an entire frame of audio and an entire frame of video are transmitted simultaneously within a period required to render the audio video stream frames by an application in a receiver. Aspects of the techniques include receiving audio and video RTP streams and assigning an entire frame of RTP video data to communication channel packets that occupy the same period, or less, as the video frame rate. Also an entire frame of RTP audio data is assigned to communication channel packets that occupy the same period, or less, as the audio frame rate. The video and audio communication channel packets are transmitted simultaneously. Receiving and assigning RTP streams can be performed in a remote station, or a base station.

Claims

1. A data stream synchronizer comprising:

a first decoder configured to receive a first encoded data stream and to output a decoded first data stream, wherein the first encoded data stream has a first bit rate during an information interval;

a second decoder configured to receive a second encoded data stream and to output a decoded second data stream, wherein the second encoded data stream has a second bit rate during the information interval;

a first buffer configured to accumulate the first decoded data stream for at least one information interval and to output a frame of the first decoded data stream each interval period;

a second buffer configured to accumulate the second decoded data stream for at least one information interval and to output a frame of the second decoded data stream each interval period; and

a combiner configured to receive the frame of first decoded data stream and the frame of second decoded data stream and to output a synchronized frame of first and second decoded data streams.

2. A data stream synchronizer as defined in claim 1, wherein the first encoded data stream is video data.

3. A data stream synchronizer as defined in claim 1, wherein the second encoded data stream is audio data.

4. A data stream synchronizer as defined in claim 1, wherein the first bit rate is higher than the second bit rate.

5. A remote station apparatus comprising:

a video decoder configured to receive encoded video data and to output decoded video data;

an audio decoder configured to receive encoded audio data and to output decoded audio data;

a video buffer configured to accumulate decoded video data for at least one frame periods and to output a frame of video data each frame period;

an audio buffer configured to accumulate decoded audio data for multiple frame periods and to output a frame of audio data each frame period; and

a combiner configured to receive the frame of video data and the frame of audio data and to output a synchronized frame of audio video data.

6. A remote station as defined in claim 5, wherein the video decoder is an MPEG decoder, H.263 decoder, or H.264 decoder.

7. A remote station as defined in claim 5, wherein the audio decoder is an MPEG decoder, H.263 decoder, or H.264 decoder.

8. A remote station as defined in claim 5, further comprising a control processor that controls the decoding and synchronization of audio and video data.

9. A remote station apparatus comprising:

a video communication channel interface configured to receive a video RTP stream and to assign an entire frame of RTP video data to communication channel packets that occupy the same period, or less, than the video frame rate;

an audio communication channel interface configured to receive an audio RTP stream and to assign an entire frame of RTP audio data to communication channel packets that occupy the same period, or less, than the audio frame rate; and

a transmitter configured to receive and transmit the video and audio communication channel packets.

10. A remote station apparatus as defined in claim 9, further comprising a control processor that controls the assignment of audio and video data to communication channel packets.

11. A base station apparatus comprising:

a video decoder configured to receive encoded video data and to output decoded video data;

an audio decoder configured to receive encoded audio data and to output decoded audio data;

a video buffer configured to accumulate decoded video data for a video frame period and to output a frame of video data each frame period;

an audio buffer configured to accumulate decoded audio data for an audio frame period and to output a frame of audio data each frame period; and

a combiner configured to receive the frame of video data and the frame of audio data and to output a synchronized frame of audio video data.

12. A base station as defined in claim 11, wherein the video decoder is an MPEG decoder, H.263 decoder, or H.264 decoder.

13. A base station as defined in claim 11, wherein the audio decoder is an MPEG decoder, H.263 decoder, or H.264 decoder.

14. A base station as defined in claim 11, further comprising a control processor that controls the decoding and synchronization of audio and video data.

15. A base station apparatus comprising:

a video communication channel interface configured to receive a video RTP stream and to assign an entire frame of RTP video data to communication channel packets that occupy the same period, or less, than the video frame rate;

an audio communication channel interface configured to receive an audio RTP stream and to assign an entire frame of RTP audio data to communication channel packets that occupy the same period, or less, than the audio frame rate; and

a transmitter configured to receive and transmit the video and audio communication channel packets.

16. A base station apparatus as defined in claim 15, further comprising a control processor that controls the assignment of audio and video data to communication channel packets.

17. A wireless communication system comprising:

a base station apparatus comprising;

a video communication channel interface configured to receive a video RTP stream and to assign an entire frame of RTP video data to communication channel packets that occupy the same period, or less, than the video frame rate;

an audio communication channel interface configured to receive an audio RTP stream and to assign an entire frame of RTP audio data to communication channel packets that occupy the same period, or less, than the audio frame rate;

a transmitter configured to receive and transmit the video and audio communication channel packets;

a remote station apparatus comprising;

a video decoder configured to receive video communication channel packets and to output decoded video data;

an audio decoder configured to receive audio communication channel packets and to output decoded audio data;

a video buffer configured to accumulate decoded video data for a video frame period and to output a frame of video data each frame period;

an audio buffer configured to accumulate decoded audio data for an audio frame period and to output a frame of audio data each frame period; and

a combiner configured to receive the frame of video data and the frame of audio data and to output a synchronized frame of audio video data.

18. A wireless communication system comprising:

a remote station apparatus comprising;

a video communication channel interface configured to receive a video RTP stream and to assign an entire frame of RTP video data to communication channel packets that occupy the same period, or less, than the video frame rate;

an audio communication channel interface configured to receive an audio RTP stream and to assign an entire frame of RTP audio data to communication channel packets that occupy the same period, or less, than the audio frame rate;

a transmitter configured to receive and transmit the video and audio communication channel packets;

a base station apparatus comprising;

a video decoder configured to receive video communication channel packets and to output decoded video data;

an audio decoder configured to receive audio communication channel packets and to output decoded audio data;

a video buffer configured to accumulate decoded video data for a video frame period and to output a frame of video data each frame period;

an audio buffer configured to accumulate decoded audio data for an audio frame period and to output a frame of audio data each frame period; and

a combiner configured to receive the frame of video data and the frame of audio data and to output a synchronized frame of audio video data.

19. A method for decoding synchronizing data streams comprising:

receiving a first encoded data stream, decoding and outputting a decoded first data stream, wherein the first encoded data stream has a first bit rate during an information interval;

receiving a second encoded data stream, decoding and outputting a decoded second data stream, wherein the second encoded data stream has a second bit rate during the information interval;

accumulating the first decoded data stream for at least one information interval and outputting a frame of the first decoded data stream each interval period;

accumulating the second decoded data stream for at least one information interval and outputting a frame of the second decoded data stream each interval period; and

combining the frame of first decoded data stream and the frame of second decoded data stream and outputting a synchronized frame of first and second decoded data streams.

20. A method for decoding and synchronizing audio and video data, the method comprising:

receiving encoded video data and outputting decoded video data;

receiving encoded audio data and outputting decoded audio data;

accumulating decoded video data for a video frame period and outputting a frame of video data each frame period;

accumulating decoded audio data for an audio frame period and outputting a frame of audio data each frame period; and

combining the frame of video data and the frame of audio data and outputting a synchronized frame of audio video data every video frame period.

21. A method for encoding audio and video data, the method comprising:

receiving a video RTP stream and assigning an entire frame of RTP video data to communication channel packets that occupy the same period, or less, as a video frame rate; and

receiving an audio RTP stream and assigning an entire frame of RTP audio data to communication channel packets that occupy the same period, or less, as an audio frame rate.

22. A computer readable media embodying a method for decoding and synchronizing data streams, the method comprising:

receiving a first encoded data stream, decoding and outputting a decoded first data stream, wherein the first encoded data stream has a first bit rate during an information interval;

receiving a second encoded data stream, decoding and outputting a decoded second data stream, wherein the second encoded data stream has a second bit rate during the information interval;

accumulating the first decoded data stream for at least one information interval and outputting a frame of the first decoded data stream each interval period;

accumulating the second decoded data stream for at least one information interval and outputting a frame of the second decoded data stream each interval period; and

combining the frame of first decoded data stream and the frame of second decoded data stream and outputting a synchronized frame of first and second decoded data streams.

23. A computer readable media embodying a method for decoding and synchronizing audio and video data, the method comprising:

receiving encoded video data and to outputting decoded video data;

receiving encoded audio data and to outputting decoded audio data;

accumulating decoded video data for a video frame period and outputting a frame of video data each frame period;

accumulating decoded audio data for an audio frame period and outputting a frame of audio data each frame period; and

combining the frame of video data and the frame of audio data and outputting a synchronized frame of audio video data.

24. A computer readable media embodying a method for encoding audio and video data, the method comprising:

receiving a video RTP stream and assigning an entire frame of RTP video data to communication channel packets that occupy the same period, or less, as a video frame rate; and

receiving an audio RTP stream and assigning an entire frame of RTP audio data to communication channel packets that occupy the same period, or less, as a audio frame rate.

25. A data stream synchronizer comprising:

means for decoding a first encoded data stream and to output a decoded first data stream, wherein the first encoded data stream has a first bit rate during an information interval;

means for decoding a second encoded data stream and to output a decoded second data stream, wherein the second encoded data stream has a second bit rate during the information interval;

means for accumulating the first decoded data stream for at least one information interval and to output a frame of the first decoded data stream each interval period;

means for accumulating the second decoded data stream for at least one information interval and to output a frame of the second decoded data stream each interval period; and

means for combining the frame of first decoded data stream and the frame of second decoded data stream and to output a synchronized frame of first and second decoded data streams.

26. A remote station apparatus comprising:

means for receiving encoded video data and outputting decoded video data;

means for receiving encoded audio data and outputting decoded audio data;

means for accumulating decoded video data for a video frame period and outputting a frame of video data each frame period;

means for accumulating decoded audio data for an audio frame period and outputting a frame of audio data each frame period; and

means for combining the frame of video data and the frame of audio data and outputting a synchronized frame of audio video data.

27. A remote station apparatus comprising:

means for receiving a video RTP stream and assigning an entire frame of RTP video data to communication channel packets that occupy the same period, or less, as a video frame rate; and

means for receiving an audio RTP stream and assigning an entire frame of RTP audio data to communication channel packets that occupy the same period, or less, as a audio frame rate.

28. A base station apparatus comprising:

means for receiving encoded video data and outputting decoded video data;

means for receiving encoded audio data and outputting decoded audio data;

means for accumulating decoded video data for a video frame period and outputting a frame of video data each frame period;

means for accumulating decoded audio data for an audio frame period and outputting a frame of audio data each frame period; and

means for combining the frame of video data and the frame of audio data and outputting a synchronized frame of audio video data.

29. A base station apparatus comprising:

means for receiving a video RTP stream and assigning an entire frame of RTP video data to communication channel packets that occupy the same period, or less, as a video frame rate; and

means for receiving an audio RTP stream and assigning an entire frame of RTP audio data to communication channel packets that occupy the same period, or less, as a audio frame rate.