CONTENT SYNCHRONIZATION USING WATERMARK TIMECODES

• US 20160057317A1
• Filed: 08/20/2015
• Published: 02/25/2016
• Est. Priority Date: 08/20/2014
• Status: Active Grant

• Current Assignee: Verance Corporation
• Original Assignee: Verance Corporation
• Inventors: Zhao, Jian, Winograd, Joseph M., Petrovic, Rade
• Granted Patent: US 10,354,354 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 21/16 Program or content traceability, e.g. by watermarking
• CPC Class Codes: G06T 1/0028 Adaptive watermarking, e.g. Human Visual System [HVS]-based watermarking
• CPC Class Codes: G06T 1/005 Robust watermarking, e.g. average attack or collusion attack resistant
• CPC Class Codes: G06T 1/0085 Time domain based watermarking, e.g. watermarks spread over several images
• CPC Class Codes: G06T 2201/005 Image watermarking
• CPC Class Codes: G06T 2201/0065 Extraction of an embedded watermark; Reliable detection
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: H04N 19/467 characterised by the embedded information being invisible, e.g. watermarking
• CPC Class Codes: H04N 21/00 Selective content distribution, e.g. interactive television or video on demand [VOD] real-time bi-directional transmission of motion video data H04N7/14 ; broadcast or conference over packet switching networks H04L12/18
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/4627 Rights management associated to the content security in data switching network management H04L41/28; security management or policies for network security H04L63/20; access security in wireless networks H04W12/08
• CPC Class Codes: H04N 21/8358 involving watermark protecting executable software by watermarking G06F21/16; image watermarking in general G06T1/0021; watermarks inserted in still images for transmission purposes H04N1/32144; inserting watermarks during video coding H04N19/467
• CPC Class Codes: H04N 5/08 Separation of synchronising signals from picture signals

Abstract

Methods, devices and computer program products allow improved synchronization among multiple contents. One disclosed method includes receiving a first and a second content and performing watermark extraction to recover a first timecode as part of a watermark embedded into a segment of that content. The first timecode conforms to a first type of timing system. Next, a timing system associated with the second content is determined, and a mapping is obtained that allows timecode conversion between the first and the second timing systems. Using the mapping to synchronize the presentation of the second content with that of the first content. Thus synchronization of multiple contents is properly maintained even when processing operations such as format conversion and transcoding cause metadata streams that carry timing information to be removed.

Claims

1. A method for synchronizing multiple contents, comprising:

receiving a plurality of contents including a first and a second content;

performing watermark extraction operations on the first content to recover a first timecode that is incorporated as part of a watermark payload into a first segment of the first content, the first timecode conforming to a first type of timing system;

determining a timing system associated with the second content;

upon a determination that the timing system associated with the second content is a second timing system that is different from the first timing system, obtaining a mapping that allows one or both of;

a conversion of timecodes from the first timing system to the second timing system or a conversion of timecodes from the second timing system to the first timing system; and

synchronizing presentation of the second content with the first content using the mapping.

2. The method of claim 1, wherein the first timecode is a particular counter value that identifies the first segment of the first content relative to a reference location within the content'"'"'s timeline.

3. The method of claim 1, wherein the second timing system provides timing information according to one of the following timecode formats or protocols:

a coordinated universal time (UTC),

a linear timecode (LTC),

a vertical interval timecode (VITC),

a precision time protocol (PTP), a

network time protocol (NTP), or

a global positioning system (GPS).

4. The method of claim 1, wherein determining the second timing system includes analyzing a metadata stream associated with the second content to discern a type of timing system associated with timecodes of the second content.

5. The method of claim 1, wherein determining the second timing system includes extracting a watermark message from the second content to discern a type of timing system associated with timecodes of the second content.

6. The method of claim 1, wherein the first content and the second content are different components of a multimedia content.

7. The method of claim 6, wherein:

the first content is an audio component and the second content is a video component,

the first content is an video component and the second content is an audio component,

the first content is a television program and the second content is an advertisement,

the first content is a main audiovisual program and the second content is an interactive content, or

the first content is a main audiovisual program and the second content is a second language content.

8. The method of claim 1, wherein the first content is a main content received at a consumer electronic device and the second content is a secondary or interactive content associated with the main content.

9. The method of claim 8, wherein the first content and the second content are delivered to the consumer electronic device through different communication channels.

10. The method of claim 1, wherein the synchronizing comprises using the mapping to convert a third timecode that conforms to the second timing system to a fourth timecode that conforms to the first timing system, and presenting the portion of the second content associated with the third timecode in synchrony with the presentation of the portion of the first content associated with the fourth timecode.

11. The method of claim 1, wherein the first timecode and the second timecode are generated based on a first clock and a second clock, respectively, and the mapping includes converting the first clock to the second clock, or the second clock to the first clock.

12. The method of claim 1, wherein the mapping is obtained from an external database that is identified based on at least a portion of the extracted watermark payload.

13. The method of claim 1, wherein

the first timing system is a fixed timing system that is based on a reference timeline that begins from an initially embedded watermarked segment of the first content and increments for each subsequent watermarked segment of the first content, and

the mapping is a changeable mapping that allows conversion between the first timing system and a changeable second timing system.

14. The method of claim 1, further comprising obtaining one or more additional mappings that allow conversion of the first or the second timecodes to one or more additional timing systems associated with one or more additional contents so as to allow the one or more additional contents to be synchronized with the first segment or a subsequent segment of the first content, or with the corresponding section of the second content.

15. A device, comprising:

a processor, and

a memory including processor executable code, the processor executable code, when executed by the processor configures the device to;

receive a plurality of contents including a first and a second content;

perform watermark extraction operations on the first content to recover a first timecode that is incorporated as part of a watermark payload into a first segment of the first content, the first timecode conforming to a first type of timing system;

determine a timing system associated with the second content;

upon a determination that the timing system associated with the second content is a second timing system that is different from the first timing system, obtain a mapping that allows one or both of;

a conversion of timecodes from the first timing system to the second timing system or a conversion of timecodes from the second timing system to the first timing system; and

synchronize presentation of the second content with the first content using the mapping.

16. The device of claim 15, wherein the first timecode is a particular counter value that identifies the first segment of the first content relative to a start location within the content'"'"'s timeline.

17. The device of claim 15, wherein the second timing system conforms to one of the following timecode formats or protocols:

a linear timecode (LTC),

a vertical interval timecode (VITC),

a precision time protocol (PTP), a

network time protocol (NTP), or

a global positioning system (GPS).

18. The device of claim 15, wherein the processor executable code, when executed by the processor configures the device to analyze a metadata stream associated with the second content to discern a type of timing system associated with timecodes of the second content.

19. The device of claim 15, wherein the processor executable code, when executed by the processor configures the device to extract a watermark message from the second content to discern a type of timing system associated with timecodes of the second content.

20. The device of claim 15, wherein the first content and the second content are different components of a multimedia content.

21. The device of claim 20, wherein:

the first content is an audio component and the second content is a video component,

the first content is an video component and the second content is an audio component,

the first content is a television program and the second content is an advertisement,

the first content is a main audiovisual program and the second content is an interactive content, or

the first content is a main audiovisual program and the second content is a second language content.

22. The device of claim 15, wherein the first content is a main content received at the device and the second content is a secondary or interactive content associated with the main content.

23. The device of claim 22, wherein the device is configured to receive the first content and the second content through different communication channels.

24. The device of claim 15, wherein the processor executable code, when executed by the processor configures the device to use the mapping to convert a third timecode that conforms to the second timing system to a fourth timecode that conforms to the first timing system, and present the portion of the second content associated with the third timecode in synchrony with the presentation of the portion of the first content associated with the fourth timecode.

25. The device of claim 15, wherein the first timecode and the second timecode are generated based on a first clock and a second clock, respectively, and the mapping includes converting the first clock to the second clock, or the second clock to the first clock.

26. The device of claim 15, wherein the mapping is obtained from an external database that is identified based on at least a portion of the extracted watermark payload.

27. The device of claim 15, wherein

the first timing system is a fixed timing system that is based on a reference timeline that begins from an initially embedded watermarked segment of the first content and increments for each subsequent watermarked segment of the first content, and

the mapping is a changeable mapping that allows conversion between the first timing system and a changeable second timing system.

28. The device of claim 15, wherein the processor executable code, when executed by the processor further configures the device to obtain one or more additional mappings that allow conversion of the first or the second timecodes to one or more additional timing systems associated with one or more additional contents so as to allow the one or more additional contents to be synchronized with the first segment or a subsequent segment of the first content, or with the corresponding section of the second content.

29. A computer program product, embodied on one or more non-tangible computer readable media, comprising:

program code for receiving a plurality of contents including a first and a second content;

program code for performing watermark extraction operations on the first content to recover a first timecode that is incorporated as part of a watermark payload into a first segment of the first content, the first timecode conforming to a first type of timing system;

program code for determining a timing system associated with the second content;

program code for upon a determination that the timing system associated with the second content is a second timing system that is different from the first timing system, obtaining a mapping that allows one or both of;

a conversion of timecodes from the first timing system to the second timing system or a conversion of timecodes from the second timing system to the first timing system; and

program code for synchronizing presentation of the second content with the first content using the mapping.