Distributed media stream synchronization control

• US 9,532,099 B2
• Filed: 03/24/2015
• Issued: 12/27/2016
• Est. Priority Date: 03/24/2015
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Lawrence, Sean J., Angadimani, Raghavendra
• Primary Examiner(s): HSIA, SHERRIE Y
• Application Number:
  US14/667,525
• Publication Number:
  US 20160286260A1
• Time in Patent Office: 644 Days
• Field of Search: 348/515, 348/512, 348/500, 348/464, 348/473, 348/476
• US Class Current: 1/1
• CPC Class Codes: H04N 21/2368 Multiplexing of audio and video streams
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/4341 Demultiplexing of audio and video streams
• CPC Class Codes: H04N 21/436 Interfacing a local distribution network, e.g. communicating with another STB or one or more peripheral devices inside the home
• CPC Class Codes: H04N 21/6131 involving transmission via a mobile phone network wireless downlink channel access H04W74/006
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56

Abstract

One or more system, apparatus, method, and computer readable media is described below for wireless display synchronization of audio and video data streams received through a direct wireless link between a content source and sink. In some embodiments, the presentation time stamp (PTS) associated with the encoding of a digital audio and digital video data stream into a compressed packetized data stream at the source is utilized as a control point for synchronization of the digital audio and video stream payloads presented at the sink. In some embodiments, one or more wireless display-synchronized PTS values are determined based on a feedback signal indicative of display synchronization error. A source including an optical camera may generate audio/video (A/V) streams that encoded and packetized by a multiplexer with wireless display-synchronized PTS values. In further embodiments, a media sync user input interface is provided at one or more of the source and sink.

Claims

1. A video source apparatus, comprising:

a stream multiplexer to assign a presentation timestamp (PTS) to one or more first packets of encoded video data, and to one or more first packets of encoded audio data;

a wireless transmitter coupled to an output of the multiplexer to wirelessly stream the first packets to a paired display device; and

a PTS adjustment module coupled to an input of the multiplexer to trigger a PTS modification for one or more second packets of the encoded video and audio data that is responsive to an audio/video (A/V) presentation synchronization error perceived or determined based on an A/V output of the display device rendered from the first packets.

2. The apparatus of claim 1, wherein the PTS adjustment module is to:

receive a synchronization error signal indicative of the A/V presentation synchronization error; and

determine, based on the synchronization error signal, the PTS modification that counters the A/V presentation synchronization error.

3. The apparatus of claim 2, wherein:

the PTS adjustment module is to;

map the synchronization error signal to a predetermined PTS offset value;

generate a modified PTS value by adding the predetermined PTS offset value to a reference PTS value; and

the multiplexer is to assign the modified PTS value to one or more of the second packets of the encoded video and audio data.

4. The apparatus of claim 2, wherein:

the multiplexer is to assign a modified PTS including a PTS offset value to delay presentation of second packets of the encoded video in response to an A/V presentation synchronization error indicative of a lag in audio output relative to video output rendered based on the first packets.

5. The apparatus of claim 1, further comprising:

a synchronization error user interface to receive a user input indicative of the A/V presentation synchronization error; and

wherein the PTS adjustment module is to;

map the user input to a predetermined PTS offset value;

generate a modified PTS value by adding the predetermined PTS offset value to a reference PTS value; and

wherein the multiplexer is to;

assign a modified PTS based on a PTS offset value that is to delay presentation of second packets of the encoded video in response to an A/V presentation synchronization error indicative of a lag in audio output relative to video output rendered based on the first packets; and

assign a modified PTS based on a PTS offset value that is to delay presentation of second packets of the encoded audio in response to an A/V presentation synchronization error indicative of a lead in audio output relative to video output rendered based on the first packets.

6. The apparatus of claim 1, further comprising:

a camera module to generate digital video and audio data;

a video encoder and audio encoder coupled between an output of the camera module and an input of the multiplexer, the encoders to generate the encoded video and audio data.

7. The apparatus of claim 1, further comprising:

a video encoder and audio encoder coupled to an input of the multiplexer, the encoders to generate the encoded video and audio data;

an A/V decoder coupled to an input of the video and audio encoders, wherein the A/V decoder, video encoder, and audio encoder comprise a transcoder to receive a first encoded A/V stream and output second encoded audio and video bit streams.

8. A wireless display system, comprising:

the video source apparatus of claim 1; and

a display device to communicate the A/V presentation synchronization error over a back channel maintained between the A/V source device and the display device.

9. The wireless display system of claim 8, wherein the display apparatus further comprises a synchronization error user interface to receive a user input indicative of the A/V presentation synchronization error.

10. A wireless video display, comprising:

a wireless receiver to receive a stream of first packets from a paired source device;

a de-multiplexer coupled to an output of the wireless receiver, and to de-multiplex a compressed audio data payload from a compressed video data payload, based at least in part, on one or more presentation timestamp (PTS) associated with the first packets;

an audio decoder and video decoder coupled to an output of the de-multiplexer, the decoders to generate digital audio and video data;

an audio rendering pipeline and video rendering pipeline coupled to the decoders to output an audio/visual (A/V) representation of the digital audio and video data;

a synchronization error user interface to receive a user input indicative of a synchronization error in the A/V presentation; and

a wireless transmitter to communicate an indication of the A/V presentation synchronization error to the paired source device.

11. The display of claim 10, wherein the wireless transmitter is to send a signal indicative of the A/V presentation error over a back channel maintained with the paired source device.

12. A method for controlling synchronization of audio/video (A/V) presentation by a wireless display, the method comprising:

assigning a presentation timestamp (PTS) to one or more first packets of encoded video data, and to one or more first packets of encoded audio data;

wirelessly streaming the first packets to a paired display device;

assigning a display-synchronized PTS to one or more second packets of the encoded video and audio data that is responsive to an A/V presentation synchronization error perceived or determined based on an A/V output of the display device rendered from the first packets; and

wirelessly streaming the second packets to the paired display device.

13. The method of claim 12, further comprising:

receiving a synchronization error signal indicative of the A/V presentation synchronization error; and

determining, based on the synchronization error signal, the display-synchronized PTS that counters the A/V presentation synchronization error.

14. The method of claim 13, wherein determining the display-synchronized PTS further comprises:

mapping the synchronization error signal to a predetermined PTS offset value;

generating a modified PTS value by adding the predetermined PTS offset value to a reference PTS value; and

assigning the modified PTS value to one or more of the second packets of the encoded video and audio data associated with the reference PTS value.

15. The method of claim 12, wherein assigning a display-synchronized PTS further comprises:

assigning a modified PTS based on a PTS offset value that delays presentation of second packets of the encoded video in response to an A/V presentation synchronization error indicative of a lag in audio output relative to video output rendered based on the first packets; and

assigning a modified PTS based on a PTS offset value that delays presentation of second packets of the encoded audio in response to an A/V presentation synchronization error indicative of a lead in audio output relative to video output rendered based on the first packets.

16. The method of claim 12, further comprising:

presenting a synchronization error user interface;

receiving, through the synchronization error user interface, a user input indicative of the A/V presentation synchronization error;

mapping the user input to a predetermined PTS offset value;

generating a modified PTS by adding the predetermined PTS offset value to a reference PTS; and

assigning the modified PTS to delay presentation of second packets of the encoded video in response to an A/V presentation synchronization error indicative of a lag in audio output relative to video output rendered based on the first packets, or to delay presentation of second packets of the encoded audio in response to an A/V presentation synchronization error indicative of a lead in audio output relative to video output rendered based on the first packets.

17. The method of claim 12, further comprising:

generating the digital video and digital audio data with a camera module including an image sensor;

and encoding the digital video and digital audio data into a compressed format.

18. The method of claim 12, further comprising:

communicating the A/V presentation synchronization error over a back channel maintained between the A/V source device and the display device.

19. One or more non-transitory computer readable media including instruction stored thereon, which when executed by a processing system, cause the system to perform a method comprising:

assigning a presentation timestamp (PTS) to one or more first packets of encoded video data, and to one or more first packets of encoded audio data;

wirelessly streaming the first packets to a paired display device;

assigning a display-synchronized PTS to one or more second packets of the encoded video and audio data that is responsive to an audio/video (A/V) presentation synchronization error perceived or determined based on an A/V output of the display device rendered from the first packets; and

wirelessly streaming the second packets to the paired display device.

20. The media of claim 19, further including instructions stored thereon, which when executed by the processing system, cause the system to perform a method comprising:

assigning a modified PTS based on a PTS offset value that delays presentation of second packets of the encoded video in response to an A/V presentation synchronization error indicative of a lag in audio output relative to video output rendered based on the first packets; and

assigning a modified PTS based on a PTS offset value that delays presentation of second packets of the encoded audio in response to an A/V presentation synchronization error indicative of a lead in audio output relative to video output rendered based on the first packets.

21. A method for controlling synchronization of audio/video (A/V) presentation by a wireless display, the method comprising:

wirelessly receiving a stream of first packets from a paired source device;

de-multiplexing a compressed audio data payload from a compressed video data payload, based at least in part, on one or more presentation timestamp (PTS) associated the first packets;

decoding the compressed audio and video data payloads into digital audio and video data;

rendering an A/V representation of the digital audio and video data;

receiving a user input indicative of a synchronization error in the A/V presentation; and

wirelessly transmitting an indication of the A/V presentation synchronization error to the paired source device.

22. The method of claim 21, wherein wirelessly transmitting the indication further comprises sending a signal indicative of the A/V presentation error over a back channel maintained with the paired source device.