Real time network adaptive low latency transport stream muxing of audio/video streams for miracast

• US 9,819,604 B2
• Filed: 07/31/2013
• Issued: 11/14/2017
• Est. Priority Date: 07/31/2013
• Status: Active Grant

• Current Assignee: NVIDIA Corporation
• Original Assignee: NVIDIA Corporation
• Inventors: Gowda, Rahul, Lapicque, Olivier, Meier, Thomas
• Primary Examiner(s): FOUD, HICHAM B
• Application Number:
  US13/955,965
• Publication Number:
  US 20150036695A1
• Time in Patent Office: 1,567 Days
• Field of Search: None
• CPC Class Codes: H04L 47/19 at layers above the network layer network arrangements for networked applications for scheduling or organising the servicing of application requests H04L67/60
• CPC Class Codes: H04L 47/283 in response to processing delays, e.g. caused by jitter or round trip time [RTT]
• CPC Class Codes: H04L 47/32 by discarding or delaying data units, e.g. packets or frames
• CPC Class Codes: H04L 47/56 implementing delay-aware scheduling
• CPC Class Codes: H04N 21/23608 Remultiplexing multiplex streams, e.g. involving modifying time stamps or remapping the packet identifiers
• CPC Class Codes: H04N 21/2368 Multiplexing of audio and video streams
• CPC Class Codes: H04N 21/43615 Interfacing a Home Network, e.g. for connecting the client to a plurality of peripherals home Audio Video Interoperability [HAVI] data switching networks H04L12/2805
• CPC Class Codes: H04N 21/43637 involving a wireless protocol, e.g. Bluetooth, RF or wireless LAN [IEEE 802.11] arrangements for wireless networking or broadcasting of information in indoor or near-field type systems H04B10/114
• CPC Class Codes: H04N 21/64792 Controlling the complexity of the content stream, e.g. by dropping packets intermediate media network packet handling H04L65/765; proxy provisioning conversion or adaptation for reducing the amount or size of exchanged application data H04L67/5651; negotiation of resources in wireless networks H04W28/16

Abstract

Systems and methods for multiplexing audio/video data and generating transport streams for WiFi network with reduced latency for real time playback at a remote device. A virtual presentation clock reference (PCR) representing a scheduled transmission time of a transport stream packet at a transport stream multiplexer is calculated based on the network transmission rate and generation of the data packets. The virtual PCR is compared with the corresponding system PCR to derive a time difference. Based on the time difference, the transport stream multiplexer is configured to adaptively drop packets or throttle packet generation so as to synchronize the playback of audio/video data on a sink device with the generation of interleaved audio/video packets.

Claims

1. A method of transmitting an audio/video mixed signal over a communication network, said method comprising:

accessing a packetized stream comprising audio payloads and video payloads;

assigning said audio payloads in a audio queue and said video payloads in a video queue;

converting said packetized stream into a transport stream comprising a plurality of packets that comprise interleaved audio packets and video packets, wherein each of said plurality of packets comprises a transport header and a payload;

prior to transmission of said plurality of packets through said communication network, calculating a virtual clock reference based on a transmission bandwidth through said communication network with respect to said plurality of packets, wherein said virtual clock reference represents a scheduled time for transmitting a packet of said plurality of packets;

prior to transmission of said plurality of packets through said communication network, deriving a time difference between said virtual clock reference and a system clock reference with respect to transmission of said plurality of packets through said communication network; and

prior to transmission of said plurality of packets through said communication network and upon a determination that said time difference indicates that said virtual clock reference falls behind said system clock reference by at least a threshold, selectively dropping a number of payloads until said virtual clock reference and said system clock reference are determined to be synchronous.

2. The method of claim 1, wherein said selectively dropping comprises exclusively dropping payloads from selected video packets before dropping a payload from a selected audio packet.

3. The method of claim 2, wherein said selectively dropping further comprises one of avoiding dropping a payload from a video packet that comprises a reference video frame, and avoiding dropping a payload from a packet that comprises a packetized elementary stream (PES) header.

4. The method of claim 1 further comprising incrementing said virtual clock reference each time a payload is dropped or a packet is transmitted, and incrementing said system clock reference each time a packet is transmitted.

5. The method of claim 1 further comprising, upon a determination that said time difference indicates that said virtual clock reference is ahead of said system clock reference by at least said threshold, suspending transmission of said plurality of packets.

6. The method of claim 5, wherein said suspending transmission comprises causing a scheduler thread to enter a sleep mode for a predetermined time, or causing said scheduler thread to wait for a subsequent payload submission event.

7. The method of claim 1 further comprising, upon a determination that said time difference indicates said virtual clock reference is synchronous with said system clock reference, sending said transport stream less said number of payloads to an output buffer assigned to a client socket, and transmitting said transport stream from said output buffer to a remote display device through said communication network.

8. The method of claim 7 further comprising sending a physical or a virtual null packet while said audio queue and said video queue are empty, and incrementing said virtual clock reference accordingly.

9. The method of claim 7, wherein said transmitting said transport stream comprises transmitting said transport stream from said output buffer before said output buffer becomes full.

10. The method of claim 7 further comprising selectively dropping additional payloads upon a determination that an output buffer has no capacity to buffer incoming packets.

11. The method of claim 10, wherein said selective dropping additional payloads comprises, upon a determination that there is adequate time to acquire an additional output buffer, avoiding dropping a payload and waiting for an additional output buffer to become available.

12. A non-transitory computer-readable storage medium comprising instructions for transmitting signals over a wireless network, wherein said instructions, when executed by a processor, cause said processor to perform a method of:

receiving a packetized elementary stream (PES) comprising audio payloads and video payloads;

enqueuing said audio payloads in a audio queue and said video payloads in a video queue;

inserting a header to said PES;

packetizing said PES into a plurality of transport packets that comprise multiplexed audio packets and video packets;

wherein each of said plurality of transport packets comprises a transport header and a payload;

prior to transmission of said plurality of transport packets through said communication network, calculating a virtual clock reference based on a transmission bandwidth of said wireless network, wherein said virtual clock reference represents a scheduled time for transmitting a packet of said plurality of transport packets;

prior to transmission of said plurality of transport packets through said communication network, deriving a time difference between said virtual clock reference and a system clock reference;

prior to transmission of said plurality of transport packets through said communication network and upon a determination that said time difference indicates that said virtual clock reference falls behind said system clock reference by at least a threshold, selectively dropping a number of payloads until said virtual clock reference and said system clock reference is determined to be synchronous.

13. The non-transitory computer-readable storage medium of claim 12, wherein said method further comprises:

upon a determination that said time difference indicates that said virtual clock reference is synchronous with said system clock reference, sending transport packets to an output buffer associated with a network interface circuit; and

upon a determination that said time difference indicates that said virtual clock reference is ahead of said system clock reference by at least a threshold, pausing sending transport packets to said output buffer.

14. The non-transitory computer-readable storage medium of claim 13, wherein said selective dropping comprises precluding a packet comprising said header or a reference video frame from being dropped;

and dropping payloads from selected video packets before dropping a payload from a selected audio packet.

15. The non-transitory computer-readable storage medium of claim 13, wherein said method further comprises:

incrementing said virtual clock reference each time a payload is dropped or a packet is transmitted; and

incrementing said system clock reference each time a packet is transmitted.

16. The non-transitory computer-readable storage medium of claim 13, wherein said method further comprises:

sending a virtual null packet that takes up no transmission bandwidth while said audio queue and said video queue are empty; and

incrementing said virtual clock reference.

17. The non-transitory computer-readable storage medium of claim 13, wherein said transmitting said transport stream comprises transmitting said transport stream before said output buffer becomes full.

18. A device operable to transmit audio/video payloads to a remote computing device through a communication network, said device comprising:

a processor;

a network circuit enabling said device to access with said communication network;

a memory coupled with said network circuit, said memory operable to store instructions that, when executed by said processor, perform a method of;

accessing a packetized stream comprising audio payloads and video payloads;

enqueuing said audio payloads in a audio queue and said video payloads in a video queue;

inserting a header to said packetized stream;

packetizing said packetized stream into a plurality of transport packets that comprise multiplexed audio packets and video packets, wherein each of said plurality of transport packets comprises a transport header and a payload;

prior to transmission of said plurality of transport packets through said communication network, calculating a virtual clock based on a transmission bandwidth of said communication network, wherein said virtual clock represents a scheduled time for transmitting said packet of said plurality of transport packets;

prior to transmission of said plurality of transport packets through said communication network, deriving a time difference between said virtual clock and a system clock with respect to transmission of said plurality of transport packets; and

prior to transmission of said plurality of transport packets through said communication network and upon a determination that said time difference indicates that said virtual clock falls behind said system clock by at least a threshold latency, selectively dropping a number of payloads until said virtual clock and said system clock is determined to be synchronous.

19. The device of claim 18, wherein said method further comprises:

upon a determination that said time difference indicates that said virtual clock is synchronous with said system clock, sending transport packets to an output buffer associated with a network interface circuit; and

upon a determination that said time difference indicates that said virtual clock is ahead than said system clock by at least a threshold latency, pause sending transport packets to said output buffer.

20. The device of claim 19, wherein said selective dropping a number of payloads comprises:

precluding a packet comprising said header or a reference video frame from being dropped; and

dropping payloads from selected video packets before dropping a payload from a selected audio packet.

21. The device of claim 19, wherein said method further comprises incrementing said virtual clock each time a payload is dropped or a packet is transmitted, and incrementing said system clock each time a packet is transmitted.

22. The device of claim 19, wherein said method further comprises sending a virtual null packet that consumes no transmission bandwidth while said audio queue and said video queue are empty, and incrementing said virtual clock.