Temporal synchronization of video and audio signals

• US 7,886,151 B2
• Filed: 01/21/2003
• Issued: 02/08/2011
• Est. Priority Date: 01/22/2002
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Purdue Research Foundation (Purdue University)
• Inventors: Delp, Edward J., Lin, Eugene T.
• Primary Examiner(s): Pyzocha; Michael
• Assistant Examiner(s): Gelagay; Shewaye
• Application Number:
  US10/349,322
• Publication Number:
  US 20030221106A1
• Time in Patent Office: 2,940 Days
• Field of Search: 713/176
• US Class Current: 713/176
• CPC Class Codes: G06T 1/0028 Adaptive watermarking, e.g. Human Visual System [HVS]-based watermarking
• CPC Class Codes: G06T 1/0064 Geometric transfor invariant watermarking, e.g. affine transform invariant
• CPC Class Codes: G06T 2201/0051 Embedding of the watermark in the spatial domain
• CPC Class Codes: G06T 2201/0052 Embedding of the watermark in the frequency domain
• CPC Class Codes: H04N 19/467 characterised by the embedded information being invisible, e.g. watermarking
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/242 Synchronization processes, e.g. processing of PCR [Program Clock References] arrangements for synchronising broadcast or distribution via plural systems in broadcast distribution systems H04H20/18
• CPC Class Codes: H04N 21/426 Internal components of the client ; Characteristics thereof H04N5/44 takes precedence
• CPC Class Codes: H04N 21/8358 involving watermark protecting executable software by watermarking G06F21/16; image watermarking in general G06T1/0021; watermarks inserted in still images for transmission purposes H04N1/32144; inserting watermarks during video coding H04N19/467
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56

Abstract

A protocol for temporal synchronization of media signals with temporal components is used for digital watermarking and other applications. The synchronization protocol achieves initial synchronization by finding an initial synchronization key through analysis of a temporal media signal stream. It then uses features of the stream and a queue of one or more keys from previous frames to derive subsequent keys to maintain synchronization. If synchronization is lost due to channel errors or attacks, for example, the protocol uses the initial synchronization key to re-establish synchronization. In digital watermarking applications, the synchronization protocol is agnostic to the watermark embedding and reading functions.

Claims

1. A method of temporal synchronization of a media signal with a temporal component, the method comprising:

using a programmed computer to perform the acts of;

establishing keys relative to temporal locations in the media signal to provide temporal synchronization;

deriving perceptual features from perceptual image or audio data in temporal locations of the media signal relative to the locations of the keys;

based on the perceptual features, deriving changes to the keys to compute derived keys for other temporal locations in the media signal than the locations corresponding to the keys, wherein the keys and the derived keys represent a synchronization protocol for the media signal, and the derived keys include at least a first derived key that is a function of another key for a different temporal location of the temporal locations and a first perceptual feature of the perceptual features.

2. The method of claim 1 wherein the synchronization protocol is used to control digital watermark embedding of an auxiliary hidden signal in the media signal as a function of the keys and the derived keys.

3. The method of claim 1 wherein a state machine takes the first perceptual feature and one or more previous keys as input, and generates output for establishing a subsequent key.

4. The method of claim 3 wherein a synchronization state in the state machine is periodic over time in the media signal.

5. The method of claim 1 wherein the synchronization protocol is used to control digital watermark reading of an auxiliary hidden signal in the media signal as a function of the keys and the derived keys.

15. The method of claim 1 wherein the first perceptual feature is a perceptual hash of an image or audio signal in the media signal.

6. A non-transitory computer readable medium on which is stored instructions, which when executed by a computer, perform a method of temporal synchronization of a media signal with a temporal component, the method comprising:

establishing keys relative to temporal locations in the media signal to provide temporal synchronization;

deriving perceptual features from perceptual image or audio data in temporal locations of the media signal relative to the locations of the keys;

based on the perceptual features, deriving changes to the keys to compute derived keys for other temporal locations in the media signal than the locations corresponding to the keys, wherein the keys and the derived keys represent a synchronization protocol for the media signal, and the derived keys include a first derived key that is a function of another key for a different temporal location of the temporal locations and a first perceptual feature of the perceptual features.

7. A temporal synchronization method for digital watermarking comprising:

using a programmed computer to perform the acts of;

computing at least one perceptual feature from an image or audio signal representing perceptual content in the time varying portions of a host signal;

using a state machine to vary a watermarking key over time as a function of the at least one perceptual feature and a key for a different temporal location; and

using the watermarking key to generate a digital watermark signal for use in embedding a digital watermark into a host signal or detecting a digital watermark from a host signal;

wherein the state machine varies the watermarking key dependent on the at least one perceptual feature of the host signal.

8. The method of claim 7 wherein the state machine includes randomization.

9. The method of claim 8 wherein the randomization is derived from a random function.

10. The method of claim 8 wherein the state machine includes a cryptographic hash function.

11. The method of claim 7 including:

deriving feature values from a host signal; and

using the features values as input to the state machine to generate the watermark key.

12. The method of claim 11 wherein the features are derived from a perceptual function.

13. The method of claim 12 wherein the perceptual function comprises a perceptual hash of the host signal.

14. A temporal synchronization method for digital watermarking comprising:

using a programmed computer to perform the acts of;

computing at least one perceptual feature from an image or audio signal representing perceptual content in the time varying portions of a host signal;

using a state machine to vary a watermarking key over time as a function of the at least one perceptual feature; and

using the watermarking key to generate a digital watermark signal for use in embedding a digital watermark into a host signal or detecting a digital watermark from a host signal;

wherein the state machine varies the watermarking key dependent on the at least one perceptual feature of the host signal;

wherein the state machine changes the watermarking key over time by making state transitions over time, and wherein the state transitions are dependent on variations in features of the host signal.

16. A non-transitory computer readable medium on which is stored instructions, which when executed by a computer, perform a temporal synchronization method for digital watermarking comprising:

computing at least one perceptual feature from an image or audio signal representing perceptual content in the time varying portions of a host signal;

using a state machine to vary a watermarking key over time as a function of the at least one perceptual feature and a key for a different temporal location; and

using the watermarking key to generate a digital watermark signal for use in embedding a digital watermark into a host signal or detecting a digital watermark from a host signal;

wherein the state machine varies the watermarking key dependent on the at least one perceptual feature of the host signal.