Audio and video buffer synchronization based on actual output feedback

• US 20060161835A1
• Filed: 01/20/2005
• Published: 07/20/2006
• Est. Priority Date: 01/20/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Panabaker, Ruston J.D., Chandley, Adrian
• Granted Patent: US 7,657,829 B2
• Time in Patent Office: Days
• US Class Current: 715/203
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: H04N 21/4135 external recorder interface circuits between an apparatus for recording television signals and a television receiver H04N5/775
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/4325 by playing back content from the storage medium reproduction of recorded television signals H04N5/76; reproduction of recorded television signals H04N9/79
• CPC Class Codes: H04N 21/43615 Interfacing a Home Network, e.g. for connecting the client to a plurality of peripherals home Audio Video Interoperability [HAVI] data switching networks H04L12/2805
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/44231 Monitoring of peripheral device or external card, e.g. to detect processing problems in a handheld device or the failure of an external recording device configuring of peripheral devices in general G06F9/4411; reporting information sensed by appliance or service execution status of appliance services in a home automation network H04L12/2823; monitoring the status of connected device in data switched networks H04L43/0817
• CPC Class Codes: H04N 21/6332 directed to client
• CPC Class Codes: H04N 21/654 Transmission by server directed to the client

Abstract

A method and system for keeping endpoints such as speakers and displays synchronized via feedback based on the actual output of the endpoints. A source of audiovisual content transmits corresponding digital data to one or more endpoints, such as over a home network, where it may be buffered and/or decoded for playback. Microphones or the like sense actual (post-buffering/decoding) output from one or more endpoints and feed it back to a synchronization mechanism. The synchronization mechanism employs pattern matching or similar techniques to determine whether and how to adjust the timing of endpoints to synchronize their actual outputs. Synchronization may be accomplished by controllably delaying transmission and/or other processing, by controllably changing the rate of advancing in a buffer, and/or by jumping ahead in a buffer. The synchronization mechanism may adjust multiple endpoints, e.g., when limited buffer size limits the amount of adjustment a single device can provide.

Claims

1. In a system for outputting audio and/or video content, a method comprising:

providing audio and/or video data to a first endpoint for buffering for subsequent output;

detecting actual output from the first endpoint;

adjusting, based on the actual output detected from the first endpoint, the relative timing of further actual output from the first endpoint with output corresponding to a second endpoint such that that actual output from the first endpoint is substantially synchronized with the actual output of the second endpoint.

2. The method of claim 1 wherein the first endpoint is remote from a source of the audio and/or video data, and wherein providing the audio and/or video data to the first endpoint comprises transmitting the data.

3. The method of claim 2 further comprising encoding the audio and/or video data prior to transmitting the data.

4. The method of claim 1 further comprising delaying the output corresponding to the second endpoint, and wherein adjusting the relative timing of the further actual output from the first endpoint comprises advancing in a buffer corresponding to the second endpoint faster than advancing in a buffer corresponding to the first endpoint.

5. The method of claim 1 wherein adjusting the relative timing of further actual output from the first endpoint with output corresponding to a second endpoint comprises adjusting both the actual outputs of the first and second endpoints towards one another.

6. The method of claim 1 wherein detecting the actual output from the first endpoint comprises receiving signals from at least one microphone.

7. The method of claim 1 wherein detecting the actual output from the first endpoint comprises receiving signals that are imperceptible to a typical human observer.

8. A computer-readable medium having computer-executable instructions which when executed perform the method of claim 1.

9. In a system for outputting audio and/or video content, a subsystem comprising:

a source that provides audio and/or video data corresponding to the content to endpoints for output;

a plurality of endpoints, including a first endpoint that plays the content based on received data from the source that is buffered;

a sensor that detects actual output of the first endpoint; and

a synchronization mechanism coupled to the sensor that determines synchronization of at least two endpoints including the first endpoint based on the actual output detected at the sensor, and that adjusts the relative timing of further actual output from the first endpoint with output corresponding to at least one other endpoint such that that actual output from the first endpoint is substantially synchronized with the actual output of the other endpoint.

10. The subsystem of claim 9 wherein at least one endpoint is coupled to the source via network connection, and wherein the synchronization mechanism adjusts a network delay to adjust the relative timing of further actual output from at least one endpoint.

11. The subsystem of claim 9 further comprising an external adjustment mechanism that adjusts the relative timing of further actual output from at least one endpoint.

12. The subsystem of claim 9 wherein the synchronization mechanism controls an artificial delay buffer to adjust the relative timing of further actual output from at least one endpoint.

13. The subsystem of claim 9 further comprising a transmission mechanism coupled to the source, and wherein at least one endpoint receives and buffers data received from the source via the transmission mechanism.

14. The subsystem of claim 9 wherein the audio and/or video data corresponding to the content is pre-encoded.

15. The subsystem of claim 9 further comprising an encoder at the source that encodes the audio and/or video data corresponding to the content prior to providing the audio and/or video data to at least one endpoint.

16. In an environment in which data corresponding to audio and/or video content is provided to a remote endpoint and another endpoint for output, a system comprising:

means for sensing the actual output of the remote endpoint relative to the other endpoint; and

means for adjusting the actual output of the remote endpoint relative to the actual output of the other endpoint to substantially synchronize the actual outputs.

17. The system of claim 16 wherein the means for adjusting the actual output of the remote endpoint relative to the actual output of the other endpoint includes delay means coupled to at least one of the endpoints.

18. The system of claim 16 wherein at least one of the endpoints includes a buffer, and wherein the means for adjusting the actual output of the remote endpoint relative to the actual output of the other endpoint includes means for controllably advancing in the buffer.

19. The system of claim 16 wherein the means for adjusting the actual output of the remote endpoint relative to the actual output of the other endpoint comprises pattern matching means.

20. The system of claim 16 wherein the means for adjusting the actual output of the remote endpoint relative to the actual output of the other endpoint comprises means for gradually bringing the actual outputs of the first and second endpoints into substantial synchronization.