Quality evaluation of multimedia delivery in cloud environments

• US 10,623,789 B1
• Filed: 05/26/2017
• Issued: 04/14/2020
• Est. Priority Date: 03/14/2011
• Status: Active Grant

• Current Assignee: Vmware LLC (Broadcom, Inc.)
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: Spracklen, Lawrence Andrew, Agrawal, Banit, Bidarkar, Rishi
• Primary Examiner(s): Taylor, Joshua D
• Application Number:
  US15/606,684
• Time in Patent Office: 1,054 Days
• CPC Class Codes: H04N 21/233 Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/234 Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
• CPC Class Codes: H04N 21/242 Synchronization processes, e.g. processing of PCR [Program Clock References] arrangements for synchronising broadcast or distribution via plural systems in broadcast distribution systems H04H20/18
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4622 Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859
• CPC Class Codes: H04N 21/6156 specially adapted to the upstream path of the transmission network
• CPC Class Codes: H04N 21/631 Multimode Transmission, e.g. transmitting basic layers and enhancement layers of the content over different transmission paths or transmitting with different error corrections, different keys or with different transmission protocols
• CPC Class Codes: H04N 21/8358 involving watermark protecting executable software by watermarking G06F21/16; image watermarking in general G06T1/0021; watermarks inserted in still images for transmission purposes H04N1/32144; inserting watermarks during video coding H04N19/467

Abstract

Methods, systems, and computer programs for measuring quality of multimedia delivery to a client are presented. A method includes operations for embedding video markers in a video stream of a multimedia stream, and embedding audio markers in an audio stream of the multimedia stream. The video stream and the audio stream are then transmitted separately to the client. Further, video markers received at the client are extracted from the transmitted video stream, and audio markers received at the client are extracted from the transmitted audio stream. A measure of the audio-video synchronization quality is obtained by determining a quantifiable time difference between the video stream and the audio stream received at the client, where the quantifiable time difference is calculated based on the extracted video markers and the extracted audio markers.

Claims

1. A method comprising:

receiving, at a client device, a separate video stream and an audio stream of multimedia content provided for presentation on a desktop view of the client device by a virtual desktop executing on a remote host computer;

presenting the received video stream and audio stream in the desktop view of the client device;

extracting one or more video markers encoded in the video stream;

extracting one or more audio markers encoded in the audio stream;

determining audio-video synchronization data for a plurality of video frames, wherein determining audio-video synchronization comprises;

for each of a plurality of video frames;

determining a video timestamp of a marker for the video frame;

determining an audio timestamp of an audio marker for the video frame based on PN codes from the audio stream, comprising;

correlating the received audio stream with the reference PN code to locate the PN codes in the audio stream, using positive and negative correlations of the PN codes in the audio stream relative to the reference PN code to determine respective timestamp bits, and combining timestamp bits to form the audio timestamp; and

determining a difference between the video timestamp and the audio timestamp for the video frame to calculate a difference in audio-video synchronization; and

providing the determined audio-video synchronization data determined for the plurality of video frames to a remote performance system, the audio-video synchronization data indicative of audio-video synchronization for at least a portion of the stream delivery to the client device.

2. The method of claim 1, wherein calculating a video frame associated with a particular audio marker comprises:

obtaining a frame rate for the video stream and using the frame rate and the audio timestamp to determine which frame should correspond to the audio marker.

3. The method of claim 1, wherein a particular frame is associated with a fractional frame number based on a particular frame rate for the video stream, and wherein a timestamp for a particular frame is calculated using the fractional frame number including calculating an amount of time required to deliver that fraction of the frame at the frame rate and using that amount of time to determine a timestamp for the frame.

4. The method of claim 1, wherein the audio timestamp values include a rollover counter based on a repeat time for the audio timestamps.

5. The method of claim 1, wherein out of order audio timestamps are detected and corrected based on a comparison with other determined audio timestamps.

6. The method of claim 1, wherein an audio timestamp with an unexpected wrap-around value is detected, the audio timestamp is discarded from analysis.

7. The method of claim 1, wherein in response to providing the determined synchronization data to the performance system, subsequent portions of the multimedia content are provided with corrected synchronization.

8. The method of claim 1, wherein determining a timestamp for an audio marker comprises:

generating the reference PN code that corresponds to a PN code used to embed the audio markers in the audio stream; and

comparing the generated PN code against the received audio stream to determine correlations.

9. The method of claim 1, wherein the one or more video markers are extracted from a mirror buffer that holds a copy in memory of a frame buffer for the received video stream.

10. A system comprising:

a computer including at least one memory and at least one processor, the computer configured to interact with one or more servers and to perform operation comprising;

receiving, at a client device, a separate video stream and an audio stream of multimedia content provided for presentation on a desktop view of the client device by a virtual desktop executing on a remote host computer;

presenting the received video stream and audio stream in the desktop view of the client device;

extracting one or more video markers encoded in the video stream;

extracting one or more audio markers encoded in the audio stream;

determining audio-video synchronization data for a plurality of video frames, wherein determining audio-video synchronization comprises;

for each of a plurality of video frames;

determining a video timestamp of a marker for the video frame;

determining an audio timestamp of an audio marker for the video frame based on PN codes from the audio stream, comprising;

correlating the received audio stream with the reference PN code to locate the PN codes in the audio stream, using positive and negative correlations of the PN codes in the audio stream relative to the reference PN code to determine respective timestamp bits, and combining timestamp bits to form the audio timestamp; and

determining a difference between the video timestamp and the audio timestamp for the video frame to calculate a difference in audio-video synchronization; and

providing the determined audio-video synchronization data determined for the plurality of video frames to a remote performance system, the audio-video synchronization data indicative of audio-video synchronization for at least a portion of the stream delivery to the client device.

11. The system of claim 10, wherein calculating a video frame associated with a particular audio marker comprises:

obtaining a frame rate for the video stream and using the frame rate and the audio timestamp to determine which frame should correspond to the audio marker.

12. The system of claim 10, wherein a particular frame is associated with a fractional frame number based on a particular frame rate for the video stream, and wherein a timestamp for a particular frame is calculated using the fractional frame number including calculating an amount of time required to deliver that fraction of the frame at the frame rate and using that amount of time to determine a timestamp for the frame.

13. The system of claim 10, wherein the audio timestamp values include a rollover counter based on a repeat time for the audio timestamps.

14. The system of claim 10, wherein out of order audio timestamps are detected and corrected based on a comparison with other determined audio timestamps.

15. The system of claim 10, wherein an audio timestamp with an unexpected wrap-around value is detected, the audio timestamp is discarded from analysis.

16. The system of claim 10, wherein in response to providing the determined synchronization data to the performance system, subsequent portions of the multimedia content are provided with corrected synchronization.

17. The system of claim 10, wherein determining a timestamp for an audio marker comprises:

generating the reference PN code that corresponds to a PN code used to embed the audio markers in the audio stream; and

comparing the generated PN code against the received audio stream to determine correlations.

18. The system of claim 10, wherein the one or more video markers are extracted from a mirror buffer that holds a copy in memory of a frame buffer for the received video stream.

19. A method comprising:

obtaining a multimedia stream at a virtual desktop, wherein the multimedia stream is separated into a video stream and an audio stream;

embedding video markers in video frames of the video stream;

embedding audio markers in the audio stream, the audio markers including a timestamp indicating a time elapsed from a start of the multimedia stream, wherein embedding audio markers inserting one or more timestamps in the audio stream, wherein each timestamp is represented by a sequence of bits and each bit is represented by a respective PN code instance, and wherein inserting a particular timestamp includes sequentially inserting a plurality of PN codes into the audio stream to represent each bit of the timestamp including inserting a positive version of the PN code to embed a timestamp bit of the sequence of bits with a first logical value and inserting a negative version of the PN code to embed a timestamp bit of the sequence of bits with a second logical value; and

transmitting the video stream and the audio stream separately to a desktop view window presented on a client device.

20. The method of claim 19, wherein embedding video markers includes:

inserting each video marker in a frame of the video stream, the video marker including a preconfigured sequence of pixels and being followed by a frame number encoded in pixel data.

21. The method of claim 19, wherein embedding the audio markers further includes:

embedding video frame numbers of video frames associated with the audio markers.

22. The method of claim 19, wherein each video marker including a frame number of a corresponding video frame.

23. The method of claim 19, further comprising:

receiving audio-video synchronization data from the client device, the audio-video synchronization data being determined by the client device based on the video and audio markers; and

determining a level of de-synchronization between the video stream and the audio stream for one or more video frames based on the audio-video synchronization data.

24. The method of claim 19, further comprising:

receiving audio-video synchronization data from each of a plurality of clients and using the received audio-video synchronization data to generate system level audio-video synchronization performance metrics.