Customized playback at sink device in wireless display system

• US 9,167,296 B2
• Filed: 10/02/2012
• Issued: 10/20/2015
• Est. Priority Date: 02/28/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Wang, Xiaodong, Shaukat, Fawad, Raveendran, Vijayalakshmi R.
• Primary Examiner(s): Perungavoor, Sath V
• Assistant Examiner(s): Hilaire, Clifford
• Application Number:
  US13/633,344
• Publication Number:
  US 20130223538A1
• Time in Patent Office: 1,113 Days
• Field of Search: 370/217, 370/221, 370/225
• US Class Current: 1/1
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G09G 2370/16 Use of wireless transmission of display information
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/43637 involving a wireless protocol, e.g. Bluetooth, RF or wireless LAN [IEEE 802.11] arrangements for wireless networking or broadcasting of information in indoor or near-field type systems H04B10/114
• CPC Class Codes: H04N 21/44004 involving video buffer management, e.g. video decoder buffer or video display buffer

Abstract

This disclosure describes techniques to improve a user experience in a Wireless Display (WD) system. The WD system includes a source device that provides media data to one or more sink devices. The techniques are directed toward reducing end-to-end latency in the WD system while improving video playback quality at the sink devices. More specifically, the techniques include customized playback at the sink devices based on the type of media data received from the source device. If the media data only includes video data and does not include audio data, a processing pipeline of the sink device is configured to render the video data without waiting to synchronize with non-existent audio data. If the media data includes both video data and audio data, an audio rendering start-up timer is reduced and the processing pipeline renders synchronized audio and video data according to the reduced start-up timer.

Claims

1. A method comprising:

establishing a communication session at a sink device with a source device in a Wireless Display (WD) system;

receiving, with the sink device, media data from the source device, wherein the media data includes at least video data;

decoding the received media data with a processing pipeline of the sink device;

detecting whether the decoded media data includes audio data; and

when the decoded media data does not include audio data, performing accelerated rendering of the video data with the processing pipeline of the sink device without waiting to synchronize the video data with audio data.

4. The method of claim 1, wherein performing accelerated rendering of the video data comprises increasing a sample time of the video data received from the source device based on a media time applicable for a renderer included in the processing pipeline, and rendering the video data according to the increased sample time.

5. The method of claim 1, wherein detecting whether the received media data includes audio data comprises detecting one of a presence or an absence of an audio time stamp in the received media data, wherein the presence of the audio time stamp indicates that the media data includes audio data.

6. The method of claim 1, further comprising:

receiving stream header information from the source device during a capability negotiation period for the communication session prior to receiving the media data;

configuring the processing pipeline of the sink device based on the stream header information; and

after the processing pipeline is configured, notifying the source device to begin transmitting the media data.

7. The method of claim 1, wherein the processing pipeline of the sink device includes a decoder that is not programmable, further comprising inserting dummy frames into the video data at the sink device to push the decoded samples of the video data out of the decoder for rendering.

8. The method of claim 1, further comprising, when the received media data includes audio data:

reducing an audio rendering start-up timer for a renderer included in the processing pipeline of the sink device;

synchronizing the video data with the audio data; and

rendering the synchronized audio data and video data according to the reduced start-up timer.

9. The method of claim 8, wherein configuring the processing pipeline based on the stream header information comprises configuring one or more of buffer sizes, rendering start-up timers, synchronization wait timers, and programmable decoder settings.

10. The method of claim 1, wherein the processing pipeline of the sink device includes a programmable decoder, further comprising configuring the programmable decoder to output the decoded samples of the video data in decoding order for rendering as soon as decoding is complete.

11. The method of claim 1, further comprising, upon receiving the media data from the source device, flushing one or more samples of the received media data out of a decoder included in the processing pipeline of the sink device prior to rendering the received media data, wherein the one or more samples comprise stalled samples of old media data from the source device.

12. The method of claim 1, further comprising transmitting feedback information describing transport channel conditions to the source device, wherein the feedback information is used to modify a processing pipeline of the source device.

13. The method of claim 1, wherein establishing the communication session at the sink device comprises receiving advertisements of the media data for the communication session from the source device, and sending a request to join the communication session to the source device.

2. The method of 1, wherein performing accelerated rendering of the video data comprises disabling synchronization between the video data and audio data at a renderer included in the processing pipeline.

3. The method of 1, wherein performing accelerated rendering of the video data comprises configuring a renderer included in the processing pipeline to render the video data.

14. A sink device comprising:

a processing pipeline including one or more processing units configured to establish a communication session at the sink device with a source device in a Wireless Display (WD) system, receive media data from the source device, wherein the media data includes at least video data, decode the received media data, detect whether the received media data includes audio data, and, when the received media data does not include audio data, perform accelerated rendering of the video data without waiting to synchronize the video data with audio data; and

a pipeline manager configured to manage the processing pipeline of the sink device.

16. The sink device of claim 14, wherein, when the received media data does not include audio data, the pipeline manager configures a renderer included in the processing pipeline to render the video data.

17. The sink device of claim 14, wherein the pipeline manager increases a sample time of the video data received from the source device based on a media time applicable for a renderer included in the processing pipeline, and wherein the one or more processing units of the processing pipeline are configured to render the video data according to the increased sample time.

18. The sink device of claim 14, wherein the one or more processing units of the processing pipeline are configured to detect one of a presence or an absence of an audio time stamp in the received media data, wherein the presence of the audio time stamp indicates that the media data includes audio data.

19. The sink device of claim 14,

wherein the one or more processing units of the processing pipeline are configured to receive stream header information from the source device during a capability negotiation period for the communication session prior to receiving the media data;

wherein the pipeline manager configures the processing pipeline of the sink device based on the stream header information; and

wherein, after the processing pipeline is configured, the one or more processing units of the processing pipeline are configured to notify the source device to begin transmitting the media data.

20. The sink device of claim 19, wherein the pipeline manager configures one or more of buffer sizes, rendering start-up timers, synchronization wait timers, and programmable decoder settings for the processing pipeline.

21. The sink device of claim 14, wherein, when the received media data includes audio data, the pipeline manager reduces an audio rendering start-up timer for a renderer included in the processing pipeline of the sink device, and wherein the one or more processing units of the processing pipeline are configured to synchronize the video data with the audio data, and render the synchronized audio data and video data according to the reduced start-up timer.

22. The sink device of claim 14, further comprising a decoder that is not programmable included in the processing pipeline, wherein the one or more processing units of the processing pipeline are configured to insert dummy frames into the video data at the sink device to push the decoded samples of the video data out of the decoder for rendering.

23. The sink device of claim 14, further comprising a programmable decoder included in the processing pipeline, wherein the pipeline manager configures the programmable decoder to output the decoded samples of the video data in decoding order for rendering as soon as decoding is complete.

24. The sink device of claim 14, wherein, upon receiving the media data from the source device, the one or more processing units of the processing pipeline are configured to flush one or more samples of the received media data out of a decoder included in the processing pipeline of the sink device prior to rendering the received media data, wherein the one or more samples comprise stalled samples of old media data from the source device.

25. The sink device of claim 14, wherein the one or more processing units of the processing pipeline are configured to transmit feedback information describing transport channel conditions to the source device, wherein the feedback information is used to modify a processing pipeline of the source device.

15. The sink device of 14, wherein, when the received media data does not include audio data, the pipeline manager disables synchronization between the video data and audio data at a renderer included in the processing pipeline, and wherein the one or more processing units of the processing pipeline are configured to render the video data without waiting to synchronize with audio data.

26. The sink device of claim 15, wherein the one or more processing units of the processing pipeline are configured to receive advertisements of the media data for the communication session from the source device, and send a request to join the communication session to the source device to establish the communication session at the sink device.

27. A sink device comprising:

means for establishing a communication session at the sink device with a source device in a Wireless Display (WD) system;

means for receiving media data from the source device, wherein the media data includes at least video data;

means for decoding the received media data with a processing pipeline of the sink device;

means for detecting whether the received media data includes audio data; and

when the received media data does not include audio data, means for performing accelerated rendering of the video data with the processing pipeline of the sink device without waiting to synchronize the video data with audio data.

30. The sink device of claim 27, further comprising means for increasing a sample time of the video data received from the source device based on a media time applicable for a renderer included in the processing pipeline, and means for rendering the video data according to the increased sample time.

31. The sink device of claim 27, further comprising means for detecting one of a presence or an absence of an audio time stamp in the received media data, wherein the presence of the audio time stamp indicates that the media data includes audio data.

32. The sink device of claim 27, further comprising:

means for receiving stream header information from the source device during a capability negotiation period for the communication session prior to receiving the media data;

means for configuring the processing pipeline of the sink device based on the stream header information; and

after the processing pipeline is configured, means for notifying the source device to begin transmitting the media data.

33. The sink device of claim 27, further comprising, when the received media data includes audio data:

means for reducing an audio rendering start-up timer for a renderer included in the processing pipeline of the sink device;

means for synchronizing the video data with the audio data; and

means for rendering the synchronized audio data and video data according to the reduced start-up timer.

34. The sink device of claim 27, wherein the processing pipeline of the sink device includes a decoder that is not programmable, further comprising means for inserting dummy frames into the video data at the sink device to push the decoded samples of the video data out of the decoder for rendering.

35. The sink device of claim 27, wherein the processing pipeline of the sink device includes a programmable decoder, further comprising means for configuring the programmable decoder to output the decoded samples of the video data in decoding order for rendering as soon as decoding is complete.

36. The sink device of claim 27, further comprising, upon receiving the media data from the source device, means for flushing one or more samples of the received media data out of a decoder included in the processing pipeline of the sink device prior to rendering the received media data, wherein the one or more samples comprise stalled samples of old media data from the source device.

28. The sink device of 27, further comprising, when the received media data does not include audio data, means for disabling synchronization between the video data and audio data at a renderer included in the processing pipeline, and means for rendering the video data without waiting to synchronize with audio data.

29. The sink device of 27, further comprising, when the received media data does not include audio data, means for configuring a renderer included in the processing pipeline to render the video data.

37. A non-transitory computer-readable medium comprising instructions that when executed in a sink device cause a programmable processor to:

establish a communication session at the sink device with a source device in a Wireless Display (WD) system;

receive media data from the source device, wherein the media data includes at least video data;

decode the received media data with a processing pipeline of the sink device;

detect whether the received media data includes audio data; and

when the received media data does not include audio data, perform accelerated rendering of the video data with the processing pipeline of the sink device without waiting to synchronize the video data with audio data.

38. The non-transitory computer-readable medium of 37, wherein, when the received media data does not include audio data, the instructions cause the programmable processor to disable synchronization between the video data and audio data at a renderer included in the processing pipeline, and render the video data without waiting to synchronize with audio data.

39. The non-transitory computer-readable medium of 37, wherein, when the received media data does not include audio data, the instructions cause the programmable processor to configure a renderer included in the processing pipeline to render the video data.

40. The non-transitory computer-readable medium of 37, wherein the instructions cause the programmable processor to increase a sample time of the video data received from the source device based on a media time applicable for a renderer included in the processing pipeline, and render the video data according to the increased sample time.

41. The non-transitory computer-readable medium of 37, wherein the instructions cause the programmable processor to detect one of a presence or an absence of an audio time stamp in the received media data, wherein the presence of the audio time stamp indicates that the media data includes audio data.

42. The non-transitory computer-readable medium of 37, further comprising instructions that cause the programmable processor to:

receive stream header information from the source device during a capability negotiation period for the communication session prior to receiving the media data;

configure the processing pipeline of the sink device based on the stream header information; and

after the processing pipeline is configured, notify the source device to begin transmitting the media data.

43. The non-transitory computer-readable medium of 37, further comprising, when the received media data includes audio data, instructions that cause the programmable processor to:

reduce an audio rendering start-up timer for a renderer included in the processing pipeline of the sink device;

synchronize the video data with the audio data; and

render the synchronized audio data and video data according to the reduced start-up timer.

44. The non-transitory computer-readable medium of 37, wherein the processing pipeline of the sink device includes a decoder that is not programmable, further comprising instructions that cause the programmable processor to insert dummy frames into the video data at the sink device to push the decoded samples of the video data out of the decoder for rendering.

45. The non-transitory computer-readable medium of 37, wherein the processing pipeline of the sink device includes a programmable decoder, further comprising instructions that cause the programmable processor to configure the programmable decoder to output the decoded samples of the video data in decoding order for rendering as soon as decoding is complete.

46. The non-transitory computer-readable medium of 37, further comprising, upon receiving the media data from the source device, instructions that cause the programmable processor to flush one or more samples of the received media data out of a decoder included in the processing pipeline of the sink device prior to rendering the received media data, wherein the one or more samples comprise stalled samples of old media data from the source device.