Media Synchronization for Real-Time Streaming

• US 20170105038A1
• Filed: 10/09/2015
• Published: 04/13/2017
• Est. Priority Date: 10/09/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Subramanya, Karthik Nallapeta, Damerell, Quinn, Lacher, Michael, Mladenov, Ivan
• Granted Patent: US 10,034,036 B2
• Time in Patent Office: Days
• CPC Class Codes: G11B 27/36 Monitoring, i.e. supervising the progress of recording or reproducing for digital recording G11B20/00 and s.gr.; for monitoring, testing or measuring of TV recorders of the type covered by H04N5/76 and subgroups, see H04N17/06
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/439 Processing of audio elementary streams
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56

Abstract

In one example, a client device interacting with a streaming application server may synchronize an audio stream to a corresponding video stream. The client streaming device may have an audio channel to receive a current audio packet of an audio stream from a streaming application server. The client streaming device may have a video channel to receive a current video packet of a video stream from the streaming application server. The client streaming device may execute a media synchronization module to synchronize the audio stream to the video stream by measuring a video display time change for the video stream describing a video delay between generating video data at the streaming application server and displaying the video data at the streaming client device to a user. The client streaming device may execute an audio adjustment module to adjust an audio presentation time for the audio stream indicating when a subsequent audio packet is played to the user based on the video display time change.

Claims

1. A streaming client device, comprising:

a communication interface configured to have an audio channel to receive a current audio packet of an audio stream from a streaming application server and configured to have a video channel to receive a current video packet of a video stream from the streaming application server; and

a processing core having at least one processor configured to execute a media synchronization module to synchronize the audio stream to the video stream by measuring a video display time change for the video stream describing a video delay between generating video data at the streaming application server and displaying the video data at the streaming client device to a user and configured to execute an audio adjustment module to adjust an audio presentation time for the audio stream indicating when a subsequent audio packet is played to the user based on the video display time change.

2. The streaming client device of claim 1, wherein the video channel is configured to receive a server video record time with the current video packet indicating when the current video packet was generated at the streaming application server.

3. The streaming client device of claim 1, further comprising;

memory configured to record a client display time for the current video packet indicating when the current video packet was displayed to the user.

4. The streaming client device of claim 1, wherein the media synchronization module is further configured to subtract a server video record time from a client display time to determine the video display time change.

5. The streaming client device of claim 1, wherein the audio channel is configured to receive a server audio record time with the current audio packet indicating when the current audio packet was generated at the streaming application server.

6. The streaming client device of claim 1, further comprising;

memory configured to record a client playback time for the current audio packet indicating when the current audio packet was played to the user.

7. The streaming client device of claim 1, wherein the media synchronization module is further configured to subtract a server audio record time from a client playback time to determine an audio playback time change describing an audio delay between generating audio data at the streaming application server and playing the audio data at the streaming client device to the user.

8. The streaming client device of claim 1, wherein the media synchronization module is further configured to calculate a synchronization metric describing a delay difference between the video stream and the audio stream based on a difference between the video display time change and an audio playback time change.

9. The streaming client device of claim 1, wherein the audio adjustment module is further configured to insert a silent audio packet into the audio stream when a synchronization metric is above a delay threshold.

10. The streaming client device of claim 1, wherein the audio adjustment module is further configured to lengthen the current audio packet when a synchronization metric is above a lengthening threshold.

11. The streaming client device of claim 1, wherein the audio adjustment module is further configured to compress the current audio packet when a synchronization metric is below a compression threshold.

12. The streaming client device of claim 1, wherein the audio adjustment module is further configured to drop the current audio packet when a synchronization metric is below a drop threshold.

13. The streaming client device of claim 1, wherein the media synchronization module is configured to adjust a metric threshold set describing at least one adjustment benchmark based on a synchronization feedback review.

14. A computing device, having a memory to store a video display time change for a video stream describing a video delay between generating video data at a streaming application server and displaying the video data at the computing device to a user and an audio playback time change for an audio stream describing an audio delay between generating audio data at the streaming application server and playing the audio data at the computing device to the user, the computing device configured to receive in a client device a current video packet of the video stream from the streaming application server, the computing device further configured to receive the current audio packet of an audio stream from the streaming application server, the computing device also configured to synchronize the audio stream to the video stream by measuring the video display time change in the video stream, the computing device is further configured to calculate a synchronization metric describing a delay difference between the video stream and the audio stream based on a difference between the video display time change and an audio playback time change, and the computing device also configured to adjust an audio presentation time for the audio stream indicating when a subsequent audio packet is played to the user based on the synchronization metric.

15. The computing device of claim 14, wherein the computing device is further configured to lengthen the current audio packet when a synchronization metric is above a lengthening threshold.

16. The computing device of claim 14, wherein the computing device is further configured to drop the current audio packet when a synchronization metric is below a drop threshold.

17. The computing device of claim 14, wherein the computing device is further configured to compress the current audio packet when a synchronization metric is below a compression threshold.

18. The computing device of claim 14, wherein the computing device is further configured to insert a silent audio packet into the audio stream when a synchronization metric is above a delay threshold.

19. A machine-implemented method, comprising:

receiving, in a streaming client device, a current video packet of a video stream and a server video record time indicating when the current video packet was generated from a streaming application server;

receiving, in the streaming client device, a current audio packet of an audio stream from the streaming application server;

recording a client display time for the current video packet indicating when the current video packet was displayed to a user;

subtracting a server video record time from a client display time to determine a video display time change for a video stream describing a video delay between generating video data at the streaming application server and displaying the video data at the streaming client device to the user; and

adjusting an audio presentation time for the audio stream indicating when a subsequent audio packet is played to the user based on the video display time change.

20. The method of claim 19, further comprising:

receiving a server audio record time with the current audio packet indicating when the current audio packet was generated from the streaming application server;

recording a client playback time for the current audio packet indicating when the current audio packet was played to the user; and

subtracting the server audio record time from the client playback time to determine an audio playback time change for an audio stream describing an audio delay between generating audio data at the streaming application server and playing the audio data at the streaming client device to the user.