Method and device for generating and detecting fingerprints for synchronizing audio and video

• US 8,358,376 B2
• Filed: 02/09/2011
• Issued: 01/22/2013
• Est. Priority Date: 07/25/2003
• Status: Expired due to Term

• Current Assignee: Gracenote, Inc. (RR Donnelley & Sons Company)
• Original Assignee: Gracenote, Inc. (RR Donnelley & Sons Company)
• Inventors: Oostveen, Job Cornelis, Roberts, David K., Denissen, Adrianus, Ten Kate, Warner
• Primary Examiner(s): Tran, Trang U
• Application Number:
  US13/024,069
• Publication Number:
  US 20110128444A1
• Time in Patent Office: 713 Days
• Field of Search: 348/515, 348/512, 348/464, 348/500, 348/518, 704/216, 704/218, 704/237, 704/278, 704/500, 704/501, 704/503, 704/504
• US Class Current: 348/515
• CPC Class Codes: G10L 25/48 specially adapted for particular use
• CPC Class Codes: G11B 27/24 by sensing features on the record carrier other than the transducing track ; sensing signals or marks recorded by another method than the main recording
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 21/23892 involving embedding information at multiplex stream level, e.g. embedding a watermark at packet level
• CPC Class Codes: H04N 21/242 Synchronization processes, e.g. processing of PCR [Program Clock References] arrangements for synchronising broadcast or distribution via plural systems in broadcast distribution systems H04H20/18
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/8547 involving timestamps for synchronizing content
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56

Abstract

This invention relates to a device and a method of generating a first and a second fingerprint (102,104) usable for synchronization of at least two signals (101,103) and corresponding method and device for synchronizing two or more signals. A fingerprint pair is generated on the basis of a segment of a first signal e.g. an audio signal and of a segment of a second signal e.g. a video signal at each synchronization time point. The generated fingerprint pair(s) are stored in a database (203) and communicated or distributed to a synchronization device (303). During synchronization, fingerprint(s) of the audio signal and fingerprint(s) of the video signal to be synchronized are generated and matched against fingerprints in the database. When a match is found, the fingerprints also determine the synchronization time point, which is used to synchronize the two signals. In this way, a simple, reliable and efficient way of synchronizing at least two signals is obtained. Further, this is enabled without modifying either the first and second signal (or subsequent signals). The signals may even be distorted or changed to some extent while still enabling synchronization.

Claims

1. A method, comprising:

deriving a first fingerprint from a segment of a first signal related to a first synchronization time point;

deriving a second fingerprint from a segment of a second signal related to a second synchronization time point;

transmitting the first and second fingerprints to a synchronization device, along with the first and second synchronization time points, or a relationship between the time points, wherein the first and the second signals are not derived from each other or a common source; and

storing the first and second fingerprints in association with the first synchronization time point and the second synchronization time point, respectively, in at least one database such that at least one of the first fingerprint or the second fingerprint is searchable for comparison with a fingerprint stream derived from a third signal, the storing including refraining from embedding the first fingerprint or the second fingerprint into the first signal or the second signal.

2. The method of claim 1, wherein the transmitting further comprises:

transmitting the first and the second fingerprints to the synchronization device via the Internet.

3. The method of claim 1, wherein the segment of the first signal and/or the segment of the second signal are unambiguously related to the first and/or second synchronization time points according to a relationship between the first and/or second synchronization time points and a starting time of the segments and/or an ending time of the segments.

4. The method of claim 1, wherein the first signal is an audio signal, and the second signal is a video signal.

5. The method of claim 1, wherein the storing comprises:

storing representations of timing relationships between the first and the second synchronization time points and a reference time.

6. The method of claim 5, further comprising:

transmitting at least one of the fingerprints or the representations to the synchronization device or a server communicatively coupled to the synchronization device.

7. A non-transitory computer readable medium having stored thereon instructions for causing one or more processing units to execute a method, comprising:

generating a first fingerprint stream from a first signal;

generating a second fingerprint stream from a second signal;

comparing a segment of the first fingerprint stream with one or more first fingerprints stored in at least one database in order to determine if a match exists;

comparing a segment of the second fingerprint stream with one or more second fingerprints stored in the at least database in order to determine if a match exists; and

when a match exists for both one of the first fingerprints and one of the second fingerprints, determining a location of a first synchronization time point for the first signal and a location of a second synchronization time point for the second signal, and synchronizing the first and the second signal using the determined locations.

8. The medium of claim 7, wherein the synchronizing comprises:

delaying either the first or the second signal by an amount equal to a difference between the location of the first synchronization time point and the location of the second synchronization time point.

9. The medium of claim 7, wherein the location of the first and/or the second synchronization time point are given by an unambiguous relation with a segment of the first signal and/or a segment of a second signal, respectively.

10. The medium of claim 7, wherein the first and the second synchronization time points are the same.

11. The medium of claim 7, wherein the first and the second synchronization time points are different, and wherein the method further comprises:

when the match exists for both one of the first fingerprints and one of the second fingerprints, obtaining a first representation of a relationship between the first synchronization time point and a first time point of a reference time, obtaining a second representation of a relationship between the second synchronization time point and a second time point of said reference time, and using the first and the second time points of said reference time to synchronize the first and the second signals,

instead of determining the location of the first synchronization time point for the first signal and the location of the second synchronization time point for the second signal, and synchronizing the first and the second signal using the determined locations.

12. The medium of claim 11, wherein the method further comprises:

receiving the first and/or the second representation at a synchronization device from a server, and/or receiving the one or more first fingerprints and the one or more second fingerprints from the server.

13. The medium of claim 7, wherein the first signal is an audio signal, and the second signal is a video signal.

14. A system, comprising:

a fingerprint generator to derive a first fingerprint from a segment of a first signal related to a first synchronization time point, and to derive a second fingerprint from a segment of a second signal related to a second synchronization time point, wherein the first and the second signals are not derived from each other or a common source; and

at least one database to store the first and second fingerprints in association with the first synchronization time point and the second synchronization time point, respectively, such that at least one of the first fingerprint or the second fingerprint is searchable for comparison with a fingerprint stream derived from a third signal, the storing including refraining from embedding the first fingerprint or the second fingerprint into the first signal or the second signal.

15. The system of claim 14, further comprising:

a synchronization device to synchronize at least two signals comprising the first signal and the second signal.

16. The system of claim 15, further comprising:

a transmitter to transmit the fingerprints from at least one database to the synchronization device.

17. The system of claim 14, wherein the segment of the first signal and/or the segment of the second signal are unambiguously related to the first and/or the second synchronization time points according to a relationship between the first and/or second synchronization time points and a starting time of the segments and/or an ending time of the segments.

18. The system of claim 14, further comprising:

a transmitter to transmit a representation of the relationship to a synchronization device, or to a server communicatively coupled to the synchronization device;

or

a transmitter to transmit one or more of the fingerprints to the server.