Synchronization of content between networked devices

• US 10,880,848 B2
• Filed: 02/24/2020
• Issued: 12/29/2020
• Est. Priority Date: 12/16/2015
• Status: Active Grant

• Current Assignee: Sonos, Inc.
• Original Assignee: Sonos, Inc.
• Inventors: Keyser-Allen, Lee, Higdon, Thomas, Vega-Zayas, Luis, Lin, Ted, Sarkic, Andrej
• Primary Examiner(s): Kassim, Khaled M
• Application Number:
  US16/798,936
• Publication Number:
  US 20200260396A1
• Time in Patent Office: 309 Days
• CPC Class Codes: H04L 43/106 using time related information in packets, e.g. by adding timestamps
• CPC Class Codes: H04L 65/60 Network streaming of media packets
• CPC Class Codes: H04N 21/4307 Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
• CPC Class Codes: H04N 21/43076 of the same content streams on multiple devices, e.g. when family members are watching the same movie on different devices
• CPC Class Codes: H04N 21/43615 Interfacing a Home Network, e.g. for connecting the client to a plurality of peripherals home Audio Video Interoperability [HAVI] data switching networks H04L12/2805
• CPC Class Codes: H04N 21/4392 involving audio buffer management
• CPC Class Codes: H04N 21/8113 comprising music, e.g. song in MP3 format
• CPC Class Codes: H04R 2227/003 Digital PA systems using, e.g. LAN or internet
• CPC Class Codes: H04R 2227/005 Audio distribution systems for home, i.e. multi-room use
• CPC Class Codes: H04R 2420/07 Applications of wireless loudspeakers or wireless microphones
• CPC Class Codes: H04R 27/00 Public address systems circuits for preventing acoustic reaction H04R3/02; circuits for distributing signals to loudspeakers H04R3/12; monitoring or testing arrangements for public address systems H04R29/007; amplifiers H03F
• CPC Class Codes: H04W 56/0015 one node acting as a reference for the others

Abstract

A control device of media playback system receives a user command to initiate playback of video content and associated audio content and transmits a BLUETOOTH message based on the command to a video playback device. The video playback device receives the BLUETOOTH message, obtains the media content via a local area network (LAN), and determines an indication of when to begin playback of the media content. The video playback device transmits the indication to an audio playback device via the LAN and outputs the video content in lip-synchrony with playback of the corresponding audio content by the audio playback device. While outputting the video content in lip-synchrony with playback of the corresponding audio content by the audio playback device, the video playback device transmits timing information via the LAN to the audio playback device to maintain lip-synchrony with playback of the corresponding audio content by the audio playback device.

Claims

1. A media playback system comprising:

a control device comprising;

a network interface configured to wirelessly communicate using a BLUETOOTH communication protocol;

a user interface configured to detect input from a user;

at least one processor;

at least one non-transitory computer-readable medium;

program instructions stored on the at least one non-transitory computer-readable medium that are executable by the at least one processor such that the control device is configured to;

receive, via the user interface, a command to initiate playback of media content, wherein the media content comprises video content and audio content associated with the video content; and

transmit, via the network interface, at least one message to a video playback device based on the command; and

the video playback device comprising;

a first network interface configured to wirelessly communicate usi ng the BLUETOOTH communication protocol;

a second network interface configured to communicate over at least one local area network (LAN);

at least one processor;

at least one non-transitory computer-readable medium; and

program instructions stored on the at least one non-transitory computer-readable medium that are executable by the at least one processor such that the video playback device is configured to;

join a synchrony group with at least one audio playback device;

receive, via the first network interface, the at least one message from the control device;

obtain, via the second network interface, the media content based on the at least one message;

determine an indication of when to begin playback of the media content;

transmit, via the second network interface, the indication of when to begin playback of the media content to the at least one audio playback device;

based on the indication of when to begin playback of the media content, output the video content in lip-synchrony with playback of the corresponding audio content by the at least one audio playback device; and

while the video content is output in lip-synchrony with playback of the corresponding audio content by the at least one audio playback device, transmit, via the second network interface, timing information to the at least one audio playback device to maintain lip-synchrony with playback of the corresponding audio content by the at least one audio playback device, wherein the transmitted timing information causes the at least one audio playback device to modify a playback rate of the corresponding audio content.

2. The media playback system of claim 1, wherein the video playback device is a television comprising a display and wherein the program instructions that are executable by the at least one processor such that the video playback device is configured to output the video content comprises program instructions that are executable by the at least one processor such that the video playback device is configured to display the video content on the display.

3. The media playback system of claim 1, wherein the control device is a first control device and wherein the media playback system further comprises a second control device comprising:

a network interface configured to communicate over at least one LAN;

a user interface configured to detect input from a user;

at least one processor;

at least one non-transitory computer-readable medium; and

program instructions stored on the at least one non-transitory computer-readable medium that are executable by the at least one processor such that the second control device is configured to;

receive, via the user interface, a command to modify the synchrony group; and

cause the synchrony group to be modified based on the received command to modify the synchrony group.

4. The media playback system of claim 3, wherein the program instructions that are executable by the at least one processor such that the second control device is configured to cause the synchrony group to be modified comprises program instructions that are executable by the at least one processor such that the second control device is configured to transmit at least one message to the video playback device.

5. The media playback system of claim 4, wherein the program instructions that are executable by the at least one processor such that the video playback device is configured to join the synchrony group with the at least one audio playback device comprises program instructions that are executable by the at least one processor such that the video playback device is configured to join the synchrony group with the at least one audio playback device based on the at least one message from the second control device.

6. The media playback system of claim 1, wherein the media content comprises a plurality of frames and wherein the program instructions that are executable by the at least one processor such that the video playback device is configured to transmit the timing information comprises program instructions that are executable by the at least one processor such that the video playback device is configured to transmit at least one packet comprising an amount of time until playback of at least one of the plurality of frames.

7. The media playback system of claim 1, wherein the media content comprises a plurality of frames and wherein the program instructions that are executable by the at least one processor such that the video playback device is configured to transmit the timing information comprises program instructions that are executable by the at least one processor such that the video playback device is configured to transmit at least one packet comprising a presentation timestamp for at least one of the plurality of frames.

8. The media playback system of claim 1, wherein the video playback device further comprises program instructions that are executable by the at least one processor such that the video playback device is configured to determine a network latency for a connection between the video playback device and the at least one audio playback device and wherein the program instructions that are executable by the at least one processor such that the video playback device is configured to determine the indication of when to begin playback of the media content comprises program instructions that are executable by the at least one processor such that the video playback device is configured to determine the indication of when to begin playback of the media content based on the determined network latency.

9. The media playback system of claim 8, wherein the program instructions that are executable by the at least one processor such that the video playback device is configured to determine the indication of when to begin playback of the media content comprises program instructions that are executable by the at least one processor such that the video playback device is configured to:

send a test packet to the at least one audio playback device at a first time based on a local clock of the video playback device;

receive an acknowledgement packet from the at least one audio playback device at a second time based on the local clock of the video playback device;

determine a test packet delay calculation based on a difference between the second time and the first time; and

determine the network latency based on the test packet delay calculation.

10. The media playback system of claim 1, wherein the video playback device further comprises:

an audio amplifier; and

a speaker coupled to the audio amplifier.

11. The media playback system of claim 1, wherein the at least one audio playback device comprises a first playback device and wherein the playback system further comprises the first playback device.

12. The media playback system of claim 11, wherein the first playback device comprises:

a network interface configured to communicate over the LAN;

an audio amplifier;

a speaker coupled to the audio amplifier;

at least one processor;

at least one non-transitory computer-readable medium; and

program instructions stored on the at least one non-transitory computer-readable medium that are executable by the at least one processor such that the first playback device is configured to;

receive, via the network interface, the indication of when to begin playback of the media content from the video playback device;

based on the indication of when to begin playback of the media content, play back the corresponding audio content in lip-synchrony with the video playback device using the audio amplifier and the speaker; and

while playing back the corresponding audio content in lip-synchrony with playback of the video content by the video playback device,

receive, via the second network interface, the timing information from the video playback device; and

modify the playback rate of the corresponding audio content based on the timing information.

20. The media playback system of claim 1, wherein causing the at least one audio playback device to modify a playback rate of the corresponding audio content comprises causing the at least one audio playback device to modify a local clock rate based on the transmitted timing information.

13. A method performed by a media playback system comprising a control device and a video playback device, the method comprising:

receiving, via a user interface of the control device, a command to initiate playback of media content, wherein the media content comprises video content and audio content associated with the video content; and

transmitting, via a network interface of the control device using a BLUETOOTH communication protocol, at least one message to the video playback device based on the command;

receiving, via a first network interface of the video playback device using the BLUETOOTH communication protocol, the at least one message from the control device;

obtaining, via a second network interface of the video playback device, the media content based on the at least one message;

determining, using the video playback device, an indication of when to begin playback of the media content;

transmitting, via the second network interface of the video playback device, the indication of when to begin playback of the media content to at least one audio playback device;

based on the indication of when to begin playback of the media content, outputting, using the video playback device, the video content in lip-synchrony with playback of the corresponding audio content by the at least one audio playback device; and

while outputting the video content in lip-synchrony with playback of the corresponding audio content by the at least one audio playback device, transmitting, via the second network interface of the video playback device, timing information to the at least one audio playback device to maintain lip-synchrony with playback of the corresponding audio content by the at least one audio playback device, wherein the transmitted timing information causes the at least one audio playback device to modify a playback rate of the corresponding audio content.

14. The method of claim 13, wherein outputting the video content comprises displaying, via a display on the video device, the video content.

15. The method of claim 13, wherein the control device is a first control device and wherein the method further comprises:

receiving, via a user interface of a second control device, a command to modify a synchrony group; and

causing, via the second control device, the synchrony group to be modified based on the received command to modify the synchrony group.

16. The method of claim 15, wherein causing the synchrony group to be modified comprises transmitting, via the second control device, at least one message to the video playback device.

17. The method of claim 16, wherein joining the synchrony group with the at least one audio playback device comprises joining the synchrony group with the at least one audio playback device based on the at least one message from the second control device.

18. The method of claim 13, wherein the media content comprises a plurality of frames and wherein transmitting the timing information comprises:

transmitting, via the video playback device to the at least one audio playback device, at least one packet comprising an amount of time until playback of at least one of the plurality of frames.

19. The method of claim 13, wherein the media content comprises a plurality of frames and wherein transmitting the timing information comprises:

transmitting, via the video playback device to the at least one audio playback device, at least one packet comprising a presentation timestamp for at least one of the plurality of frames.