Frame capture and buffering at source device in wireless display system

• US 9,491,505 B2
• Filed: 10/02/2012
• Issued: 11/08/2016
• Est. Priority Date: 02/28/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Wang, Xiaodong, Shaukat, Fawad, Raveendran, Vijayalakshmi R.
• Primary Examiner(s): Faragalla, Michael
• Assistant Examiner(s): Edwards, Mark
• Application Number:
  US13/633,328
• Publication Number:
  US 20130222210A1
• Time in Patent Office: 1,498 Days
• Field of Search: 345/2.3, 455/3.01, 725/131
• US Class Current: 1/1
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G09G 2370/16 Use of wireless transmission of display information
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/43637 involving a wireless protocol, e.g. Bluetooth, RF or wireless LAN [IEEE 802.11] arrangements for wireless networking or broadcasting of information in indoor or near-field type systems H04B10/114
• CPC Class Codes: H04N 21/44004 involving video buffer management, e.g. video decoder buffer or video display buffer

Abstract

This disclosure describes techniques to improve a user experience in a Wireless Display (WD) system. The WD system includes a source device that provides media data to one or more sink devices. The techniques are directed toward reducing end-to-end latency in the WD system while improving video playback quality at the sink devices. More specifically, the techniques include low latency screen capture and buffering at the source device. For example, a processing pipeline of the source device may be configured to include minimum-size buffers between processing steps to reduce latency. The techniques include buffering a most recent frame update captured from the media data in the minimum-size buffers and dropping older frame updates when the minimum-size buffers are full. In addition, the processing pipeline may be configured to use hardware acceleration to retrieve the frame updates from the buffers for processing.

Claims

1. A method comprising:

establishing a communication session at a source device with one or more sink devices in a Wireless Display (WD) system;

capturing frame updates of media data at the source device;

upon establishing the communications session in the WD system and before processing any frame updates for transmission to the one or more sink devices, configuring a processing pipeline of the source device to include minimum-size buffers and use hardware acceleration to reduce latency in the processing pipeline, wherein the minimum-size buffers comprise at least two buffers that each have a size that is capable of holding at least a most recent one of the captured frame updates;

buffering at least the most recent one of the captured frame updates in the minimum-size buffers included in the processing pipeline of the source device;

retrieving the frame updates in the minimum-size buffers using hardware acceleration; and

processing the frame updates retrieved from the minimum-size buffers with the processing pipeline for transmission to the one or more sink devices.

2. The method of claim 1, wherein capturing frame updates of media data comprises storing frame updates in a memory of the source device.

3. The method of claim 2, wherein buffering at least a most recent one of the frame updates comprises buffering one of a memory copy of the most recent one of the frame updates stored in the memory or a pointer to the most recent one of the frame updates stored in the memory.

4. The method of claim 1, further comprising:

determining whether the minimum-size buffers are full or not full; and

buffering, upon determining that the minimum-size buffers are not full or after dropping older frame updates of the media data from the minimum-size buffers when the minimum-size buffers are full, the most recent one of the frame updates in the minimum-size buffers.

5. The method of claim 1, wherein retrieving the frame updates using hardware acceleration comprises moving the frame updates between the minimum-size buffers and processing units in the processing pipeline of the source device without using a central processing unit (CPU) of the source device.

6. The method of claim 1, wherein the minimum-size buffers comprise two write-back (WB) buffers between a multimedia display processor and a video processing engine in the processing pipeline of the source device.

7. The method of claim 1, wherein the minimum-size buffers comprise four frame buffers between a video processing engine and an encoder in the processing pipeline of the source device.

8. The method of claim 1, wherein processing the frame updates with the processing pipeline comprises:

rendering the frame updates into frame buffers for the WD system included in the processing pipeline; and

encoding the rendered frame updates for transmission to the one or more sink devices.

9. The method of claim 8, further comprising buffering the encoded frame updates prior to transmission to the one or more sink device in coded picture buffers of the source device.

10. The method of claim 9, wherein buffering the encoded frame updates comprises buffering a pointer to the encoded frame updates stored in a memory of the source device without creating a memory copy of the encoded frame update.

11. The method of claim 1, further comprising:

transmitting the processed frame updates to the one or more sink devices; and

after a period of time without receiving a new frame update, retransmitting the processed frame updates from the source device to the one or more sink devices.

12. The method of claim 1, further comprising configuring the processing pipeline of the source device to prioritize audio data transport over video data transport to the one or more sink devices.

13. The method of claim 1, further comprising:

receiving feedback information describing transport channel conditions from at least one of the sink devices; and

modifying the processing pipeline of the source device based on the feedback information.

14. The method of claim 1, wherein establishing the communication session at the source device comprises advertising the media data for the communication session to a plurality of sink devices, and receiving requests to join the communication session from the one or more sink devices.

15. A source device comprising:

a processing pipeline including one or more processing units configured to establish a communication session at the source device with one or more sink devices in a Wireless Display (WD) system, capture frame updates of the media data, buffer at least a most recent one of the captured frame updates in minimum-size buffers included in the processing pipeline, and process the frame updates from the minimum-size buffers for transmission to the one or more sink devices;

a pipeline manager configured to, upon establishing the communications session in the WD system and before processing any frame updates for transmission to the one or more sink devices, configure the processing pipeline to include the minimum-size buffers and use hardware acceleration to reduce latency in the processing pipeline, wherein the minimum-size buffers comprise at least two buffers that each have a size that is capable of holding at least the most recent one of the captured frame updates; and

a hardware accelerator configured to retrieve the frame updates from the minimum-size buffers for processing by the processing pipeline.

16. The source of claim 15, wherein the one or more processing units of the processing pipeline are configured to store frame updates in a memory of the source device.

17. The source device of claim 16, wherein the one or more processing units of the processing pipeline are configured to buffer one of a memory copy of the most recent one of the frame updates stored in the memory or a pointer to the most recent one of the frame updates stored in the memory.

18. The source device of claim 15, wherein the one or more processing units of the processing pipeline are configured to:

determine whether the minimum-size buffers are full or not full; and

buffer, upon determining that the minimum-size buffers are not full or after dropping older frame updates of the media data from the minimum-size buffers when the minimum-size buffers are full, the most recent one of the frame updates in the minimum-size buffers.

19. The source device of claim 15, wherein the hardware accelerator is configured to move the frame updates between the minimum-size buffers and processing units in the processing pipeline of the source device without using a central processing unit (CPU) of the source device.

20. The source device of claim 15, wherein the minimum-size buffers comprise two write-back (WB) buffers between a multimedia display processor and a video processing engine in the processing pipeline of the source device.

21. The source device of claim 15, wherein the minimum-size buffers comprise four frame buffers between a video processing engine and an encoder in the processing pipeline of the source device.

22. The source device of claim 15, wherein the one or more processing units of the processing pipeline are configured to:

render the frame updates into frame buffers for the WD system included in the processing pipeline; and

encode the rendered frame updates for transmission to the one or more sink devices.

23. The source device of claim 22, wherein the one or more processing units of the processing pipeline are configured to buffer the encoded frame updates prior to transmission to the one or more sink devices in coded picture buffers of the source device.

24. The source device of claim 23, wherein the one or more processing units of the processing pipeline are configured to buffer a pointer to the encoded frame updates stored in a memory of the source device without creating a memory copy of the encoded frame update.

25. The source device of claim 15, wherein the one or more processing units of the processing pipeline are configured to:

transmit the processed frame updates to the one or more sink devices; and

after a period of time without receiving a new frame update, retransmit the processed frame updates from the source device to the one or more sink devices.

26. The source device of claim 15, wherein the pipeline manager configures the processing pipeline to prioritize audio data transport over video data transport to the one or more sink devices.

27. The source device of claim 15, wherein the pipeline manager:

receives feedback information describing transport channel conditions from at least one of the sink devices; and

modifies the processing pipeline of the source device based on the feedback information.

28. The source device of claim 15, wherein the one or more processing units of the processing pipeline are configured to advertise the media data for the communication session to a plurality of sink devices, and receive requests to join the communication session from the one or more sink devices to establish the communication session at the source device.

29. A source device comprising:

means for establishing a communication session at the source device with one or more sink devices in a Wireless Display (WD) system;

means for capturing frame updates of media data;

means for, upon establishing the communications session in the WD system and before processing any frame updates for transmission to the one or more sink devices, configuring a processing pipeline of the source device to include minimum-size buffers and use hardware acceleration to reduce latency in the processing pipeline, wherein the minimum-size buffers comprise at least two buffers that each have a size that is capable of holding at least a most recent one of the captured frame updates;

means for buffering at least the most recent one of the captured frame updates in the minimum-size buffers included in the processing pipeline of the source device;

means for retrieving the frame updates from the minimum-size buffers using hardware acceleration; and

means for processing the frame updates retrieved from the minimum-size buffers with the processing pipeline for transmission to the one or more sink devices.

30. The source device of claim 29, further comprising means for storing frame updates in a memory of the source device.

31. The source device of claim 30, further comprising means for buffering one of a memory copy of the most recent one of the frame updates stored in the memory or a pointer to the most recent one of the frame updates stored in the memory.

32. The source device of claim 29, further comprising:

means for determining whether the minimum-size buffers are full or not full; and

means for buffering the most recent one of the frame updates in the minimum-size buffers upon determining that the minimum-size buffers are not full or after dropping older frame updates of the media data from the minimum-size buffers when the minimum-size buffers are full.

33. The source device of claim 29, further comprising means for moving the frame updates between the minimum-size buffers and processing units in the processing pipeline of the source device using hardware acceleration and without using a central processing unit (CPU) of the source device.

34. The source device of claim 29, further comprising:

means for rendering the frame updates into frame buffers for the WD system included in the processing pipeline; and

means for encoding the rendered frame updates for transmission to the one or more sink devices.

35. The source device of claim 34, further comprising means for buffering the encoded frame updates prior to transmission to the one or more sink devices in coded picture buffers of the source device, wherein the means for buffering the encoded frame updates comprise means for buffering a pointer to the encoded frame updates stored in a memory of the source device without creating a memory copy of the encoded frame update.

36. The source device of claim 29, further comprising:

means for transmitting the processed frame updates to the one or more sink devices; and

after a period of time without receiving a new frame update, means for retransmitting the processed frame updates from the source device to the one or more sink devices.

37. A computer-readable medium comprising instructions that when executed in a source device cause a programmable processor to:

establish a communication session at the source device with one or more sink devices in a Wireless Display (WD) system;

capture frame updates of media data at the source device;

upon establishing the communications session in the WD system and before processing any frame updates for transmission to the one or more sink devices, configure a processing pipeline of the source device to include minimum-size buffers and use hardware acceleration to reduce latency in the processing pipeline, wherein the minimum-size buffers comprise at least two buffers that each have a size that is capable of holding at least a most recent one of the captured frame updates;

buffer at least the most recent one of the captured frame updates in the minimum-size buffers included in the processing pipeline of the source device; and

process the frame updates from the minimum-size buffers with the processing pipeline for transmission to the one or more sink devices, wherein the frame updates are retrieved from the minimum-size buffers using hardware acceleration.

38. The computer-readable medium of claim 37, wherein the instructions, when executed, cause the programmable processor to store frame updates in a memory of the source device.

39. The computer-readable medium of claim 38, wherein the instructions, when executed, cause the programmable processor to buffer one of a memory copy of the most recent one of the frame updates stored in the memory or a pointer to the most recent one of the frame updates stored in the memory.

40. The computer-readable medium of claim 37, wherein the instructions, when executed, cause the programmable processor to:

determine whether the minimum-size buffers are full or not full; and

buffer, upon determining that the minimum-size buffers are not full or after dropping older frame updates of the media data from the minimum-size buffers when the minimum-size buffers are full, the most recent one of the frame updates in the minimum-size buffers.

41. The computer-readable medium of claim 37, wherein the instructions, when executed, cause the programmable processor to move the frame updates between the minimum-size buffers and processing units in the processing pipeline of the source device using hardware acceleration and without using a central processing unit (CPU) of the source device.

42. The computer-readable medium of claim 37, wherein the instructions, when executed, cause the programmable processor to:

render the frame updates into frame buffers for the WD system included in the processing pipeline; and

encode the rendered frame updates for transmission to the one or more sink devices.

43. The computer-readable medium of claim 42, further comprising instructions that, when executed, cause the programmable processor to buffer the encoded frame updates prior to transmission to the one or more sink device in coded picture buffers of the source device, wherein the instructions, when executed, further cause the programmable processor to buffer a pointer to the encoded frame updates stored in a memory of the source device without creating a memory copy of the encoded frame update.

44. The computer-readable medium of claim 37, further comprising instructions that, when executed, cause the programmable processor to:

transmit the processed frame updates to the one or more sink devices; and

after a period of time without receiving a new frame update, retransmit the processed frame updates from the source device to the one or more sink devices.