Method and device for generating and detecting fingerprints for synchronizing audio and video

• US 7,907,211 B2
• Filed: 07/20/2004
• Issued: 03/15/2011
• Est. Priority Date: 07/25/2003
• Status: Active Grant

• Current Assignee: Gracenote, Inc. (RR Donnelley & Sons Company)
• Original Assignee: Gracenote, Inc. (RR Donnelley & Sons Company)
• Inventors: Oostveen, Job Cornelis, Ten Kate, Warner Rudolph Theophile, Roberts, David Keith, Denissen, Adrianus Johannes Maria
• Primary Examiner(s): Tran; Trang U
• Application Number:
  US10/566,003
• Publication Number:
  US 20080062315A1
• Time in Patent Office: 2,429 Days
• Field of Search: 348/515, 348/512, 348/464, 348/500, 348/518, 704/216, 704/218, 704/237, 704/278, 704/500, 704/501, 704/503, 704/504
• US Class Current: 348/515
• CPC Class Codes: G10L 25/48 specially adapted for particular use
• CPC Class Codes: G11B 27/24 by sensing features on the record carrier other than the transducing track ; sensing signals or marks recorded by another method than the main recording
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 21/23892 involving embedding information at multiplex stream level, e.g. embedding a watermark at packet level
• CPC Class Codes: H04N 21/242 Synchronization processes, e.g. processing of PCR [Program Clock References] arrangements for synchronising broadcast or distribution via plural systems in broadcast distribution systems H04H20/18
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/8547 involving timestamps for synchronizing content
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56

Abstract

This invention relates to a device and a method of generating a first and a second fingerprint (102,104) usable for synchronisation of at least two signals (101,103) and corresponding method and device for synchronising two or more signals. A fingerprint pair is generated on the basis of a segment of a first signal e.g. an audio signal and of a segment of a second signal e.g. a video signal at each synchronisation time point. The generated fingerprint pair(s) are stored in a database (203) and communicated or distributed to a synchronisation device (303). During synchronisation, fingerprint(s) of the audio signal and fingerprint(s) of the video signal to be synchronised are generated and matched against fingerprints in the database. When a match is found, the fingerprints also determine the synchronisation time point, which is used to synchronise the two signals. In this way, a simple, reliable and efficient way of synchronising at least two signals is obtained. Further, this is enabled without modifying either the first and second signal (or subsequent signals). The signals may even be distorted or changed to some extent while still enabling synchronisation.

Claims

1. A computer-implemented method of synchronizing two or more signals, the method comprising:

generating a first fingerprint stream on the basis of a first signal,

generating a second fingerprint stream on the basis of a second signal,

comparing a segment of the first fingerprint stream with one or more first fingerprints stored in at least one database in order to determine if a match exists or not,

comparing a segment of the second fingerprint stream with one or more second fingerprints stored in the at least one database in order to determine if a match exists or not, and

if a match exists for both a first and a second fingerprint determining a location of a first synchronization time point (Tn, Tn+1) for the first signal and a location of a second synchronization time point (Tn, Tn+1;

Tm) for the second signal and synchronizing the first and the second signal using the determined locations.

2. A method according to claim 1, wherein synchronizing comprises:

delaying either the first or the second signal by an amount equal to a difference, if any, between the location of the first synchronization time point (Tn, Tn+1) for the first signal and the location of the second synchronization time point (Tn, Tn+1;

Tm) for the second signal.

3. A method according to claim 1, wherein the location of at least one of the first and the second synchronization time point (Tn, Tn+1;

Tm) for the first/and the second signal are given by an unambiguous relation with at least one of a segment of a first signal and a segment of a second signal used during generation of the matching first fingerprint and of the matching second fingerprint.

4. A method according to claim 1, wherein the first and second synchronization time point (Tn;

Tn+1;

Tm) is the same.

5. A method according to claim 1, wherein the first and second synchronization time point (Tn;

Tn+1;

Tm) is different and in that the method further comprises;

if a match exists for both a first and a second fingerprint

obtaining a first representation of a relationship between the first synchronization time point (Tn;

Tn+1) and a first time point of a reference time,

obtaining a second representation of a relationship between the second synchronization time point (Tn;

Tn+1;

Tm) and a second time point of said reference time, and

using the first and second time points of said reference time to synchronize the first and the second signal,

instead of

determining, if a match exists for both a first and a second fingerprint, a location of a first synchronization time point (Tn, Tn+1) for the first signal and a location of a second synchronization time point (Tn, Tn+1;

Tm) for the second signal and synchronizing the first and the second signal using the determined locations.

6. A method according to claim 5, wherein the method further comprises at least one of:

receiving at least one of the first and second representation in synchronization device from a server in communications connection with the synchronization device, and

receiving the one or more first fingerprints and second fingerprints from the server.

7. A synchronization device for synchronizing two or more signals, the device comprising:

means for generating a first fingerprint stream on the basis of a first signal,

means for generating a second fingerprint stream on the basis of a second signal,

means for comparing a segment of the first fingerprint stream with one or more first fingerprints stored in at least one database in order to determine if a match exists or not,

means for comparing a segment of the second fingerprint stream with one or more second fingerprints stored in the at least one database in order to determine if a match exists or not, and

means for, if a match exists for both a first and a second fingerprint, determining a location of a first synchronization time point (Tn;

Tn+1) for the first signal and determining a location of a second synchronization time point (Tn;

Tn+1;

Tm) for the second signal and means for synchronizing the first and the second signal using the determined locations.

8. A device according to claim 7, wherein the means for synchronizing is adapted to:

delay either the first or the second signal by an amount equal to a difference, if any, between the location of the synchronization time point (Tn;

Tn+1) for the first signal and the location of the synchronization time point (Tn;

Tn+1;

Tm) for the second signal.

9. A device according to claim 7, wherein the location of at least one of the first and second synchronization time point (Tn;

Tn+1;

Tm) for at least one of the first and second signal are given by an unambiguous relation with at least one of a segment of a first signal and a segment of a second signal used during generation of the matching first fingerprint and of the matching second fingerprint.

10. A device according to claim 7, wherein the first and second synchronization time point (Tn;

Tn+1;

Tm) is the same.

11. A device according to claim 7, wherein the first and second synchronization time point (Tn;

Tn+1;

Tm) is different and in that the device further comprises;

if a match exists for both a first and a second fingerprint,

a receiver for obtaining a first representation of a relationship between the first synchronization time point (Tn;

Tn+1) and a first time point of a reference time,

a receiver for obtaining a second representation of a relationship between the second synchronization time point (Tn;

Tn+1;

Tm) and a second time point of said reference time, and

synchronization means for using the first and second time points of said reference time to synchronize the first and the second signal,

instead of comprising

means for, if a match exists for both a first and a second fingerprint, determining a location of a first synchronization time point (Tn;

Tn+1) for the first signal and determining a location of a second synchronization time point (Tn;

Tn+1;

Tm) for the second signal and means for synchronizing the first and the second signal using the determined locations.

12. A device according to claim 11, wherein the device further comprises at least one of:

a receiver for receiving at least one of the first and second representation in a synchronization device from a server in communications connection with the synchronization device, and

a receiver for receiving the one or more first fingerprints and second fingerprints from the server.