Media synchronization for real-time streaming

• US 10,034,036 B2
• Filed: 10/09/2015
• Issued: 07/24/2018
• Est. Priority Date: 10/09/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Subramanya, Karthik Nallapeta, Lacher, Michael, Damerell, Quinn, Mladenov, Ivan
• Primary Examiner(s): Christensen, Scott B
• Application Number:
  US14/880,178
• Publication Number:
  US 20170105038A1
• Time in Patent Office: 1,019 Days
• Field of Search: None
• CPC Class Codes: G11B 27/36 Monitoring, i.e. supervising the progress of recording or reproducing for digital recording G11B20/00 and s.gr.; for monitoring, testing or measuring of TV recorders of the type covered by H04N5/76 and subgroups, see H04N17/06
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/439 Processing of audio elementary streams
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56

Abstract

In one example, a client device interacting with a streaming application server may synchronize an audio stream to a corresponding video stream. The client streaming device may have an audio channel to receive a current audio packet of an audio stream from a streaming application server. The client streaming device may have a video channel to receive a current video packet of a video stream from the streaming application server. The client streaming device may execute a media synchronization module to synchronize the audio stream to the video stream by measuring a video display time change for the video stream describing a video delay between generating video data at the streaming application server and displaying the video data at the streaming client device to a user. The client streaming device may execute an audio adjustment module to adjust an audio presentation time for the audio stream indicating when a subsequent audio packet is played to the user based on the video display time change.

Claims

1. A streaming client device, comprising:

a communication interface configured to have an audio channel to receive an audio stream of a streaming media content from a streaming application server and configured to have a video channel to receive a video stream of the streaming media content from the streaming application server,

wherein the video stream and audio stream are separately transmitted concurrently by the streaming application server to the streaming client device; and

a processing core having at least one processor configured to execute a media synchronization module to synchronize the audio stream to the video stream by measuring and storing a video display time change for each of a plurality of video packets of the video stream, wherein the video display time change is a representation of a total video delay between generating video data included in a current video packet of the plurality of video packets of the video stream at the streaming application server and displaying the video data at the streaming client device to a user,

wherein the processing core is configured to execute an audio adjustment module to adjust an audio presentation time for the audio stream by adjusting when a subsequent audio packet of the audio stream is played to the user based on the stored video display time change to synchronize playback of the subsequent audio packet with a subsequent video packet of the video stream.

2. The streaming client device of claim 1, wherein the video channel is configured to receive a server video record time with the current video packet indicating when the current video packet was generated at the streaming application server.

3. The streaming client device of claim 2, further comprising a memory configured to record a client display time for the current video packet indicating when the current video packet was displayed to the user.

4. The streaming client device of claim 3, wherein the media synchronization module is further configured to subtract the received server video record time from the recorded client display time to determine the video display time change.

5. The streaming client device of claim 1, wherein the audio channel is configured to receive a server audio record time with the current audio packet indicating when the current audio packet was generated at the streaming application server.

6. The streaming client device of claim 1, further comprising;

memory configured to record a client playback time for the current audio packet indicating when the current audio packet was played to the user.

7. The streaming client device of claim 1, wherein the media synchronization module is further configured to subtract a server audio record time from a client playback time to determine an audio playback time change describing an audio delay between generating audio data at the streaming application server and playing the audio data at the streaming client device to the user.

8. The streaming client device of claim 1, wherein the media synchronization module is further configured to calculate a synchronization metric describing a delay difference between the video stream and the audio stream based on a difference between the video display time change and an audio playback time change.

9. The streaming client device of claim 1, wherein the audio adjustment module is further configured to insert a silent audio packet into the audio stream when a synchronization metric is above a delay threshold.

10. The streaming client device of claim 1, wherein the audio adjustment module is further configured to lengthen the current audio packet when a synchronization metric is above a lengthening threshold.

11. The streaming client device of claim 1, wherein the audio adjustment module is further configured to compress the current audio packet when a synchronization metric is below a compression threshold.

12. The streaming client device of claim 1, wherein the audio adjustment module is further configured to drop the current audio packet when a synchronization metric is below a drop threshold.

13. The streaming client device of claim 1, wherein the media synchronization module is configured to adjust a metric threshold set describing at least one adjustment benchmark based on a synchronization feedback review.

14. A computing device, comprising:

a memory to store a video display time change for a video stream wherein the video display time change is a representation of a total video delay between generating video data at a streaming application server and displaying the video data at the computing device to a user and an audio playback time change for an audio stream wherein the audio display time change is a representation of a total audio delay between generating audio data at the streaming application server and playing the audio data at the computing device to the user; and

a processor configured to;

receive in the computing device a plurality of video packets of the video stream of a streaming media content from the streaming application server;

receive an audio stream of the streaming media content from the streaming application server, wherein the video stream and audio stream are separately transmitted concurrently by the streaming application server to the computing device;

synchronize the audio stream to the video stream by measuring and storing the video display time change for each of the plurality of video packets of the video stream;

calculate and store a synchronization metric that represents a delay difference between the video stream and the audio stream based on a difference between the video display time change and the audio playback time change; and

adjust an audio presentation time for the audio stream by adjusting when a subsequent audio packet of the audio stream is played to the user based on the stored synchronization metric to synchronize playback of the subsequent audio packet with a subsequent video packet of the video stream.

15. The computing device of claim 14, wherein the processor is further configured to lengthen the current audio packet when a synchronization metric is above a lengthening threshold.

16. The computing device of claim 14, wherein the processor is further configured to drop the current audio packet when a synchronization metric is below a drop threshold.

17. The computing device of claim 14, wherein the processor is further configured to compress the current audio packet when a synchronization metric is below a compression threshold.

18. The computing device of claim 14, wherein the processor is further configured to insert a silent audio packet into the audio stream when a synchronization metric is above a delay threshold.

19. A machine-implemented method, comprising:

receiving, in a streaming client device, a plurality of video packets of a video stream of a streaming media content and a server video record time indicating when each video packet was generated from a streaming application server;

receiving, in the streaming client device, a current audio packet of an audio stream of the streaming media content from the streaming application server,

wherein the video stream and audio stream are separately transmitted concurrently by the streaming application server to the streaming client device;

recording a client display time for a current video packet of the plurality of video packets indicating when the current video packet was displayed to a user;

subtracting the server video record time of the current video packet from the client display time for the current video packet to determine and store a video display time change for the video stream, wherein the video display time change is a representation of a total video delay between generating video data included in the current video packet of the plurality of video packets of the video stream at the streaming application server and displaying the video data at the streaming client device to the user; and

adjusting an audio presentation time for the audio stream by adjusting when a subsequent audio packet of the audio stream is played to the user based on the stored video display time change to synchronize playback of the subsequent audio packet with a subsequent video packet of the video stream.

20. The method of claim 19, further comprising:

receiving a server audio record time with the current audio packet indicating when the current audio packet was generated from the streaming application server;

recording a client playback time for the current audio packet indicating when the current audio packet was played to the user; and

subtracting the server audio record time from the client playback time to determine an audio playback time change for an audio stream describing an audio delay between generating audio data at the streaming application server and playing the audio data at the streaming client device to the user.