Synchronizing playback of digital media content

• US 10,382,511 B2
• Filed: 02/23/2017
• Issued: 08/13/2019
• Est. Priority Date: 02/25/2016
• Status: Active Grant

• Current Assignee: Amp Me Inc.
• Original Assignee: Amp Me Inc.
• Inventors: Archambault, Martin-Luc, Paquet, Andre-Philippe, Gagnon, Martin, Levesque, Daniel, Beaudoin, Keith, Bernard, Luc, Mercier, Marc, Mimeault, Michael, Kuret, Rami, Simard, Julien Gobeil
• Primary Examiner(s): Algibhah, Hamza N
• Application Number:
  US15/441,020
• Publication Number:
  US 20170251040A1
• Time in Patent Office: 901 Days
• CPC Class Codes: G10L 19/018 Audio watermarking, i.e. embedding inaudible data in the audio signal
• CPC Class Codes: H04L 43/106 using time related information in packets, e.g. by adding timestamps
• CPC Class Codes: H04L 65/611 for multicast or broadcast systems for broadcast or conference H04L12/18; arrangements for broadcast or distribution combined with broadcast H04H20/00; arrangements for broadcast applications with a direct linkage to broadcast information or to broadcast space-time H04H60/00; selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS], H04W4/06
• CPC Class Codes: H04L 65/75 Media network packet handling
• CPC Class Codes: H04L 65/756 adapting media to device capabilities
• CPC Class Codes: H04L 65/764 at the destination reformatting of additional data in video clients H04N21/4355
• CPC Class Codes: H04N 21/4305 Synchronising client clock from received content stream, e.g. locking decoder clock with encoder clock, extraction of the PCR packets arrangements for synchronising receiver with transmitter by comparing receiver clock with transmitter clock H04L7/0012; arrangements for synchronising receiver with transmitter wherein the receiver takes measures against momentary loss of synchronisation H04L7/0083
• CPC Class Codes: H04N 21/43076 of the same content streams on multiple devices, e.g. when family members are watching the same movie on different devices
• CPC Class Codes: H04N 21/8547 involving timestamps for synchronizing content

Abstract

A guest digital device joins a playback session hosted by a host digital device and synchronizes playback of digital media content with playback on the host digital device. This enables multiple users to listen to or view media at the same time on their respective devices. A time-based synchronization process may be used to initially synchronize playback of the guest digital device to the host digital device. If synchronization is lost, the device may record and analyze audio during playback and a fingerprint-based synchronization approach may be applied that detects a time offset between fingerprints in the analyzed audio. If synchronization is still not achieved, a watermark-based synchronization approach may be applied in which watermarks are actively injected into the audio and the time offsets between the watermarks is detected.

Claims

1. A computer-implemented method for synchronizing playback of guest media content played by a guest digital device with playback of host audio media content played on a host digital device, the method comprising:

determining, by the guest digital device, a time delta representing an offset between a local system clock of the guest digital device and a reference clock;

receiving, by the guest digital device, playback information specifying media content being played by the host digital device, a media playback start time representing a start time of the media playback on the host digital device relative to the reference clock, and a media playback session script, the media playback session script comprising a media sequence indicating a sequence of frames of the media content;

determining, based on the media playback start time, a current time of the local system clock of the guest digital device, and the media playback session script, an estimated playback position of the host digital device within a current media sequence;

determining a latency of the guest digital device representing a time between initializing playback of a media sequence and outputting the media sequence;

adjusting the estimated playback position of the guest digital device to compensate for the time delta and the latency to generate a time-adjusted playback position; and

outputting the media content on the guest digital device according to the playback session script and the time-adjusted playback position.

2. The computer-implemented method of claim 1, further comprising:

responsive to detecting a lack of synchronization between playback of the media content from the guest digital device and the host digital device;

recording, by the guest digital device, audio including ambient audio originating from the guest digital device and ambient audio originating from the host digital device;

detecting a time difference between corresponding features in the recorded audio;

adjusting the time-adjusted playback position of the media content on the guest digital device based on the time difference.

3. The computer-implemented method of claim 2, wherein detecting the time difference between the corresponding features in the record audio comprises:

generating a rolling spectrogram from the recorded audio;

analyzing the rolling spectrogram to identify time-locations of peaks in the rolling spectrogram;

determining candidate offsets from time differences between pairs of the peaks;

performing a regression to generate scores for each of the candidate offsets; and

determining a most likely offset based on the scores.

4. The computer-implemented method of claim 1, further comprising:

responsive to detecting a lack or loss of synchronization between playback of the media content from the guest digital device and the host digital device;

inserting a watermark into the media content at a given time position;

recording, by the guest digital device, audio including ambient audio originating from the guest digital device and ambient audio originating from the host digital device;

detecting a time difference between detected time locations of the watermark in the recorded audio;

adjusting the time-adjusted playback position of the media content on the guest digital device based on the time difference.

5. The computer-implemented method of claim 1, further comprising:

receiving, from the host digital device, a time adjustment in a playback position of the media content; and

adjusting the playback position of the media content on the guest digital device based on the time adjustment received from the host digital device.

6. The computer-implemented method of claim 1, wherein the watermark encodes an identifier of the media sequence in which the watermark is inserted.

7. The computer-implemented method of claim 1, wherein determining the time delta comprises:

reading a timestamp from a reference clock;

reading a timestamp from a local system clock of the guest digital device;

determining a time difference between the timestamp from the reference clock and the timestamp from the local system clock;

estimating a network latency associated with retrieving the timestamp from the reference clock; and

generating the time delta by adjusting the time difference based on the network latency.

8. The computer-implemented method of claim 1, wherein determining the estimated playback position of the host digital device within a current media sequence comprises:

determining a current session playback time;

determining a media frame duration;

determining a current frame number based on the current session playback time and the media frame duration; and

determining the estimated playback position from the current frame number.

9. A non-transitory computer-readable storage medium storing instructions for synchronizing playback of guest media content played by a guest digital device with playback of host audio media content played on a host digital device, the instructions when executed causing a processor to perform steps including:

determining a time delta representing an offset between a local system clock of the guest digital device and a reference clock;

receiving playback information specifying media content being played by the host digital device, a media playback start time representing a start time of the media playback on the host digital device relative to the reference clock, and a media playback session script, the media playback session script comprising a media sequence indicating a sequence of frames of the media content;

determining, based on the media playback start time, a current time of the local system dock of the guest digital device, and the media playback session script, an estimated playback position of the host digital device within a current media sequence;

determining a latency of the guest digital device representing a time between initializing playback of a media sequence and outputting the media sequence;

adjusting the estimated playback position of the guest digital device to compensate for the time delta and the latency to generate a time-adjusted playback position; and

outputting the media content according to the playback session script and the time-adjusted playback position.

10. The non-transitory computer, readable storage medium of claim 9, the instructions when executed further causing the processor to perform steps including:

responsive to detecting a lack of synchronization between playback of the media content from the guest digital device and the host digital device;

recording, by the guest digital device, audio including ambient audio originating from the guest digital device and ambient audio originating from the host digital device;

detecting a time difference between corresponding features in the recorded audio;

adjusting the time-adjusted playback position of the media content on the guest digital device based on the time difference.

11. The non-transitory computer-readable storage medium of claim 10, the instructions when executed further causing the processor to perform steps including:

generating a rolling spectrogram from the recorded audio;

analyzing the rolling spectrogram to identify time-locations of peaks in the rolling spectrogram;

determining candidate offsets from time differences between pairs of the peaks;

performing a regression to generate scores for each of the candidate offsets; and

determining a most likely offset based on the scores.

12. The non-transitory computer-readable storage medium of claim 9, further comprising:

responsive to detecting a lack or loss of synchronization between playback of the media content from the guest digital device and the host digital device;

inserting a watermark into the media content at a given time position;

recording, by the guest digital device, audio including ambient audio originating from the guest digital device and ambient audio originating from the host digital device;

detecting a time difference between detected time locations of the watermark in the recorded audio;

adjusting the time-adjusted playback position of the media content on the guest digital device based on the time difference.

13. The non-transitory computer-readable storage medium of claim 9, wherein the instructions when executed furthercause the processor to perform steps including:

receiving, from the host digital device, a time adjustment in a playback position of the media content; and

adjusting the playback position of the media content on the guest digital device based on the time adjustment received from the host digital device.

14. The non-transitory computer-readable storage medium of claim 9, wherein the watermark encodes an identifier of the media sequence in which the watermark is inserted.

15. The non-transitory computer-readable storage medium of claim 9, wherein determining the time delta comprises:

reading a timestamp from a reference clock;

reading a timestamp from a local system clock of the guest digital device;

determining a time difference between the timestamp from the reference clock and the timestamp from the local system clock;

estimating a network latency associated with retrieving the timestamp from the reference clock; and

generating the time delta by adjusting the time difference based on the network latency.

16. The non-transitory computer-readable storage medium of claim 9, wherein determining the estimated playback position of the host digital device within a current media sequence comprises:

determining a current session playback time;

determining a media frame duration;

determining a current frame number based on the current session playback time and the media frame duration; and

determining the estimated playback position from the current frame number.

17. A computing device for synchronizing playback of guest media content with playback of host media content played on a host digital device, the computing device comprising:

an audio playback device;

a processor; and

a non-transitory computer-readable storage medium storing instructions that when executed cause the processor to perform steps including;

determining a time delta representing an offset between a local system clock of the guest digital device and a reference clock;

receiving playback information specifying media content being played by the host digital device, a media playback start time representing a start time of the media playback on the host digital device relative to the reference dock, and a media playback session script, the media playback session script comprising a media sequence indicating a sequence of frames of the media content;

determining, based on the media playback start time, a current time of the local system clock of the guest digital device, and the media playback session script, an estimated playback position of the host digital device within a current media sequence;

determining a latency of the guest digital device representing a time between initializing playback of a media sequence and outputting the media sequence;

adjusting the estimated playback position of the guest digital device to compensate for the time delta and the latency to generate a time-adjusted playback position; and

outputting the media content to the audio playback device according to the playback session script and the time-adjusted playback position.

18. The computing device of claim 17, the instructions when executed further causing the processor to perform steps including:

responsive to detecting a lack of synchronization between playback of the media content from the guest digital device and the host digital device;

recording, by the guest digital device, audio including ambient audio originating from the guest digital device and ambient audio originating from the host digital device;

detecting a time difference between corresponding features in the recorded audio;

adjusting the time-adjusted playback position of the media content on the guest digital device based on the time difference.

19. The computing device of claim 18, the instructions when executed further causing the processor to perform steps including:

generating a rolling spectrogram from the recorded audio;

analyzing the rolling spectrogram to identify time-locations of peaks in the rolling spectrogram;

determining candidate offsets from time differences between pairs of the peaks;

performing a regression to generate scores for each of the candidate offsets; and

determining a most likely offset based on the scores.

20. The computing device of claim 17, further comprising:

responsive to detecting a lack or loss of synchronization between playback of the media content from the guest digital device and the host digital device;

inserting a watermark into the media content at a given time position;

recording, by the guest digital device, audio including ambient audio originating from the guest digital device and ambient audio originating from the host digital device;

detecting a time difference between detected time locations of the watermark in the recorded audio;

adjusting the time-adjusted playback position of the media content on the guest digital device based on the time difference.