Multiple path audio video synchronization

• US 20070220561A1
• Filed: 03/20/2006
• Published: 09/20/2007
• Est. Priority Date: 03/20/2006
• Status: Active Grant

• Current Assignee: Pixelworks Incorporated
• Original Assignee: VIXS Systems Incorporated (Pixelworks Incorporated)
• Inventors: Saleem, Shawn, Girardeau, James
• Granted Patent: US 7,907,212 B2
• Time in Patent Office: Days
• US Class Current: 725/80
• CPC Class Codes: H04N 21/2368 Multiplexing of audio and video streams
• CPC Class Codes: H04N 21/4305 Synchronising client clock from received content stream, e.g. locking decoder clock with encoder clock, extraction of the PCR packets arrangements for synchronising receiver with transmitter by comparing receiver clock with transmitter clock H04L7/0012; arrangements for synchronising receiver with transmitter wherein the receiver takes measures against momentary loss of synchronisation H04L7/0083
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/4341 Demultiplexing of audio and video streams
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/4622 Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859
• CPC Class Codes: H04N 5/60 for the sound signals

Abstract

Audio video synchronization begins by receiving an audio-video signal via a first communication path. The processing continues by capturing a representation of an audio component of the audio-video signal, wherein the audio component was rendered audible via a second communication path. The processing continues by deriving a reference representation of the audio component of the audio-video signal received via the first communication path. The processing continues by aligning a video component of the audio video-signal of the first communication path with the audio component of the audio video signal of the second communication path based on the representation of the audio component and the reference representation of the audio component.

Claims

1. A method comprises:

receiving an audio-video signal via a first communication path;

capturing a representation of an audio component of the audio-video signal, wherein the audio component was rendered audible via a second communication path;

deriving a reference representation of the audio component of the audio-video signal received via the first communication path; and

aligning a video component of the audio video-signal of the first communication path with the audio component of the audio video signal of the second communication path based on the representation of the audio component and the reference representation of the audio component.

2. The method of claim 1, wherein the capturing the representation comprises:

receiving an audible rendition of the audio component via the second communication path;

converting the audible rendition into an analog audio signal; and

converting the analog audio signal into a digitized audio signal.

3. The method of claim 2, wherein the deriving a reference representation of the audio component comprises:

decoding a digital audio file portion of the audio-video signal into a digitized reference audio signal to provide the reference representation of the audio component.

4. The method of claim 3, wherein the aligning the video component of the audio video-signal received via the first communication path with the audio component of the audio video signal of the second communication path comprises:

determining an offset between the digitized audio signal and the digitized reference audio signal; and

aligning the video component of the audio video-signal received via the first communication path with the audio component of the audio video signal of the second communication path based on the offset.

5. The method of claim 4, wherein the determining the offset comprises at least one of:

correlating at least portion a frequency range of the digitized audio signal with at least portion a frequency range of the digitized reference audio signal;

interpreting embedded frame information;

interpreting signal properties of the digitized audio signal and the digitized reference audio signal; and

interpreting at least one of peak levels, envelopes, and waveforms of the digitized audio signal and the digitized reference audio signal.

6. The method of claim 4 further comprises:

providing the offset to a transmitting source of the audio-video signal; and

adjusting transmission of the audio-video signal via the first communication path and the transmission of the audio component of the audio video signal via the second communication path based on the offset such that for subsequent audio-video signals transmitted by the transmission source, the video component of the audio video-signal transmitted via the first communication path is substantially aligned with the audio component of the audio video signal transmitted via the second communication path.

7. The method of claim 4, wherein the determining the offset comprises at least one of:

utilizing a nominal offset to transmit the audio-video signal via the first communication path and the audio component of the audio-video signal via the second communication path;

determining whether alignment differences between the video component of the audio video-signal of the first communication path and the audio component of the audio video signal of the second communication path are within a range of acceptable alignment differences; and

when the alignment differences are within the range of acceptable alignment differences, setting the offset as the nominal offset.

8. The method of claim 7 further comprises:

when the alignment differences are not within the range of acceptable alignment differences, establishing the offset based on the nominal offset and the alignment differences.

9. The method of claim 1 further comprises:

subsequent to the aligning of the video component of the audio video-signal received via the first communication path with the audio component of the audio video signal of the second communication path;

rendering the audio component of the audio-video signal audible via at least one of a remote device that receives the audio component via the first communication path and a source device that receives the audio component via the second communication path.

10. The method of claim 1 further comprises:

updating the aligning of the video component of the audio video-signal received via the first communication path with the audio component of the audio video signal of the second communication path.

11. A system comprises:

a source for providing an audio-video signal via a first communication path and an audio component of the audio-video signal via a second communication path; and

a remote device for capturing a representation of the audio component of the audio-video signal from the second communication path, wherein at least one of the source and the remote device aligns a video component of the audio video-signal provided via the first communication path with the audio component of the audio video signal provided via the second communication path based on a reference representation of the audio component of the audio-video signal provided via the first communication path and the representation of the audio component.

12. The system of claim 11, wherein the remote device further functions to capture the representation by:

receiving an audible rendition of the audio component via the second communication path;

converting the audible rendition into an analog audio signal; and

converting the analog audio signal into a digitized audio signal.

13. The system of claim 12, wherein at least one of the remote device and the source further functions to:

generate the reference representation of the audio component of the audio-video signal provided via the first communication path by decoding a digital audio file portion of the audio-video signal into a digitized reference audio signal to provide the reference representation of the audio component.

14. The system of claim 13, wherein the at least one of the remote device and the source align the video component of the audio video-signal received via the first communication path with the audio component of the audio video signal of the second communication path by:

determining an offset between the digitized audio signal and the digitized reference audio signal; and

aligning the video component of the audio video-signal received via the first communication path with the audio component of the audio video signal of the second communication path based on the offset.

15. The system of claim 14, wherein the at least one of the remote device and the source determine the offset by at least one of:

correlating at least portion a frequency range of the digitized audio signal with at least portion a frequency range of the digitized reference audio signal;

interpreting embedded frame information;

interpreting signal properties of the digitized audio signal and the digitized reference audio signal; and

interpreting at least one of peak levels, envelopes, and waveforms of the digitized audio signal and the digitized reference audio signal.

16. The system of claim 14, wherein the at least one of the remote device and the source further functions to:

provide the offset to a transmitting source of the audio-video signal; and

adjust transmission of the audio-video signal via the first communication path and the transmission of the audio component of the audio video signal via the second communication path based on the offset such that for subsequent audio-video signals transmitted by the transmission source, the video component of the audio video-signal transmitted via the first communication path is substantially aligned with the audio component of the audio video signal transmitted via the second communication path.

17. The system of claim 14, wherein the at least one of the remote device and the source further functions to determine the offset by at least one of:

utilizing a nominal offset to transmit the audio-video signal via the first communication path and the audio component of the audio-video signal via the second communication path;

determining whether alignment differences between the video component of the audio video-signal of the first communication path and the audio component of the audio video signal of the second communication path are within a range of acceptable alignment differences; and

when the alignment differences are within the range of acceptable alignment differences, setting the offset as the nominal offset.

18. The system of claim 17, wherein the at least one of the remote device and the source further functions to:

when the alignment differences are not within the range of acceptable alignment differences, establish the offset based on the nominal offset and the alignment differences.

19. The system of claim 11 further comprises:

subsequent to the aligning of the video component of the audio video-signal received via the first communication path with the audio component of the audio video signal of the second communication path;

the remote device rendering the audio component of the audio-video signal received via the first communication path audible; and

the source rendering the audio component of the audio-video signal provided via the second communication path audible.

20. The system of claim 11, wherein the at least one of the remote device and the source further functions to:

update the aligning of the video component of the audio video-signal provided via the first communication path with the audio component of the audio video signal provided via the second communication path.