Content synchronization using watermark timecodes

• US 10,354,354 B2
• Filed: 08/20/2015
• Issued: 07/16/2019
• Est. Priority Date: 08/20/2014
• Status: Active Grant

• Current Assignee: Verance Corporation
• Original Assignee: Verance Corporation
• Inventors: Zhao, Jian, Winograd, Joseph M., Petrovic, Rade
• Primary Examiner(s): Shang, Annan Q
• Application Number:
  US14/831,752
• Publication Number:
  US 20160057317A1
• Time in Patent Office: 1,426 Days
• Field of Search: 725139, 725 68, 725131, 725151
• CPC Class Codes: G06F 21/16 Program or content traceability, e.g. by watermarking
• CPC Class Codes: G06T 1/0028 Adaptive watermarking, e.g. Human Visual System [HVS]-based watermarking
• CPC Class Codes: G06T 1/005 Robust watermarking, e.g. average attack or collusion attack resistant
• CPC Class Codes: G06T 1/0085 Time domain based watermarking, e.g. watermarks spread over several images
• CPC Class Codes: G06T 2201/005 Image watermarking
• CPC Class Codes: G06T 2201/0065 Extraction of an embedded watermark; Reliable detection
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: H04N 19/467 characterised by the embedded information being invisible, e.g. watermarking
• CPC Class Codes: H04N 21/00 Selective content distribution, e.g. interactive television or video on demand [VOD] real-time bi-directional transmission of motion video data H04N7/14 ; broadcast or conference over packet switching networks H04L12/18
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/4627 Rights management associated to the content security in data switching network management H04L41/28; security management or policies for network security H04L63/20; access security in wireless networks H04W12/08
• CPC Class Codes: H04N 21/8358 involving watermark protecting executable software by watermarking G06F21/16; image watermarking in general G06T1/0021; watermarks inserted in still images for transmission purposes H04N1/32144; inserting watermarks during video coding H04N19/467
• CPC Class Codes: H04N 5/08 Separation of synchronising signals from picture signals

Abstract

Methods, devices and computer program products allow improved synchronization among multiple contents. One disclosed method includes receiving a first and a second content and performing watermark extraction to recover a first timecode as part of a watermark embedded into a segment of that content. The first timecode conforms to a first type of timing system. Next, a timing system associated with the second content is determined, and a mapping is obtained that allows timecode conversion between the first and the second timing systems. Using the mapping to synchronize the presentation of the second content with that of the first content. Thus synchronization of multiple contents is properly maintained even when processing operations such as format conversion and transcoding cause metadata streams that carry timing information to be removed.

Claims

1. A method for synchronizing multiple contents, comprising:

receiving a plurality of contents including a first and a second content;

performing watermark extraction operations on the first content to recover a first timecode that is incorporated as part of a watermark payload into a first segment of the first content, the first timecode conforming to a first type of timing system;

determining a timing system associated with the second content;

upon a determination that the timing system associated with the second content is a second timing system that is different from the first timing system, obtaining a mapping that allows one or both of;

a conversion of timecodes from the first timing system to the second timing system or a conversion of timecodes from the second timing system to the first timing system, wherein the mapping is obtained from an external database that is identified based on at least a portion of the extracted watermark payload; and

synchronizing presentation of the second content with the first content using the mapping, wherein one of the first or the second timing systems provides timing information according to a counter value and the other of the first or the second timing systems provides a timing information according to a timecode format selected from one of;

a coordinated universal time (UTC),

a linear timecode (LTC),

a vertical interval timecode (VITC),

a precision time protocol (PTP),

a network time protocol (NTP), or

a global positioning system (GPS); and

wherein;

the first timing system is a fixed timing system that is based on a reference timeline that begins from an initially embedded watermarked segment of the first content,

the mapping allows conversion between the first timing system and any of the second timing systems, and

the synchronizing comprises using the mapping to convert a particular timecode that conforms to the first timing system to another timecode that conforms to the second timing system, or to convert a particular timecode that conforms to the second timing system to another timecode that conforms to the first timing system, and to allow at least one component of the first content and at least one component of the second content to be rendered based on a common timeline and presented in a synchronized fashion.

2. The method of claim 1, wherein the first timecode is a particular counter value that identifies the first segment of the first content relative to a reference location within the content'"'"'s timeline.

3. The method of claim 1, wherein determining the second timing system includes analyzing a metadata stream associated with the second content to discern a type of timing system associated with timecodes of the second content.

4. The method of claim 1, wherein determining the second timing system includes extracting a watermark message from the second content to discern a type of timing system associated with timecodes of the second content.

5. The method of claim 1, wherein the first content and the second content are different components of a multimedia content.

6. The method of claim 5, wherein one or more of the following is true:

the first content is an audio component and the second content is a video component,

the first content is a video component and the second content is an audio component,

the first content is a television program and the second content is an advertisement,

the first content is a main audiovisual program and the second content is an interactive content, or

the first content is a main audiovisual program and the second content is a second language content.

7. The method of claim 1, wherein the first content is a main content received at a consumer electronic device and the second content is a secondary or interactive content associated with the main content.

8. The method of claim 7, wherein the first content and the second content are delivered to the consumer electronic device through different communication channels.

9. The method of claim 1, wherein the first timecode and the second timecode are generated based on a first clock and a second clock, respectively, and the mapping includes converting the first clock to the second clock, or the second clock to the first clock.

10. The method of claim 1, wherein

the reference timeline of the first timing system that begins from the initially embedded watermarked segment of the first content increments for each subsequent watermarked segment of the first content.

11. The method of claim 1, further comprising obtaining one or more additional mappings that allow conversion of the first or the second timecodes to one or more additional timing systems associated with one or more additional contents so as to allow the one or more additional contents to be synchronized with the first segment or a subsequent segment of the first content, or with the corresponding section of the second content.

12. The method of claim 1, wherein the at least one component of the first content and the at least one component of the second content are spliced together to form a rendered content.

13. A device, comprising:

a processor, and

a memory including processor executable code, the processor executable code, when executed by the processor configures the device to;

receive a plurality of contents including a first and a second content;

perform watermark extraction operations on the first content to recover a first timecode that is incorporated as part of a watermark payload into a first segment of the first content, the first timecode conforming to a first type of timing system;

determine a timing system associated with the second content;

upon a determination that the timing system associated with the second content is a second timing system that is different from the first timing system, obtain a mapping that allows one or both of;

a conversion of timecodes from the first timing system to the second timing system or a conversion of timecodes from the second timing system to the first timing system, wherein the mapping is obtained from an external database that is identified based on at least a portion of the extracted watermark payload; and

synchronize presentation of the second content with the first content using the mapping, wherein one of the first and second timing systems provides timing information according to a counter value and the other of the first or the second timing systems provides a timing information according to a timecode format selected from one of;

a coordinated universal time (UTC),

a linear timecode (LTC),

a vertical interval timecode (VITC),

a precision time protocol (PTP),

a network time protocol (NTP), or

a global positioning system (GPS); and

wherein;

the first timing system is a fixed timing system that is based on a reference timeline that begins from an initially embedded watermarked segment of the first content,

the mapping allows conversion between the first timing system and any of the second timing systems, and

the synchronizing comprises using the mapping to convert a particular timecode that conforms to the first timing system to another timecode that conforms to the second timing system, or to convert a particular timecode that conforms to the second timing system to another timecode that conforms to the first timing system, and to allow at least one component of the first content and at least one component of the second content to be rendered based on a common timeline and presented in a synchronized fashion.

14. The device of claim 13, wherein the first timecode is a particular counter value that identifies the first segment of the first content relative to a start location within the content'"'"'s timeline.

15. The device of claim 13, wherein the processor executable code, when executed by the processor configures the device to analyze a metadata stream associated with the second content to discern a type of timing system associated with timecodes of the second content.

16. The device of claim 13, wherein the processor executable code, when executed by the processor configures the device to extract a watermark message from the second content to discern a type of timing system associated with timecodes of the second content.

17. The device of claim 13, wherein the first content and the second content are different components of a multimedia content.

18. The device of claim 17, wherein:

the first content is an audio component and the second content is a video component,

the first content is a video component and the second content is an audio component,

the first content is a television program and the second content is an advertisement,

the first content is a main audiovisual program and the second content is an interactive content, or

the first content is a main audiovisual program and the second content is a second language content.

19. The device of claim 13, wherein the first content is a main content received at the device and the second content is a secondary or interactive content associated with the main content.

20. The device of claim 19, wherein the device is configured to receive the first content and the second content through different communication channels.

21. The device of claim 13, wherein the first timecode and the second timecode are generated based on a first clock and a second clock, respectively, and the mapping includes converting the first clock to the second clock, or the second clock to the first clock.

22. The device of claim 13, wherein

the reference timeline of the first timing system that begins from the initially embedded watermarked segment of the first content increments for each subsequent watermarked segment of the first content.

23. The device of claim 13, wherein the processor executable code, when executed by the processor further configures the device to obtain one or more additional mappings that allow conversion of the first or the second timecodes to one or more additional timing systems associated with one or more additional contents so as to allow the one or more additional contents to be synchronized with the first segment or a subsequent segment of the first content, or with the corresponding section of the second content.

24. The device of claim 13, wherein the at least one component of the first content and the at least one component of the second content are spliced together to form a rendered content.

25. A computer program product, embodied on one or more non-tangible computer readable media, comprising:

program code for receiving a plurality of contents including a first and a second content;

program code for performing watermark extraction operations on the first content to recover a first timecode that is incorporated as part of a watermark payload into a first segment of the first content, the first timecode conforming to a first type of timing system;

program code for determining a timing system associated with the second content;

program code for upon a determination that the timing system associated with the second content is a second timing system that is different from the first timing system, obtaining a mapping that allows one or both of;

a conversion of timecodes from the first timing system to the second timing system or a conversion of timecodes from the second timing system to the first timing system, wherein the mapping is obtained from an external database that is identified based on at least a portion of the extracted watermark payload; and

program code for synchronizing presentation of the second content with the first content using the mapping, wherein one of the first and second timing systems provides timing information according to a counter value and the other of the first or the second timing systems provides a timing information according to a timecode format selected from one of;

a coordinated universal time (UTC),

a linear timecode (LTC),

a vertical interval timecode (VITC),

a precision time protocol (PTP),

a network time protocol (NTP), or

a global positioning system (GPS); and

wherein;

the first timing system is a fixed timing system that is based on a reference timeline that begins from an initially embedded watermarked segment of the first content,

the mapping allows conversion between the first timing system and any of the second timing systems, and

the synchronizing comprises using the mapping to convert a particular timecode that conforms to the first timing system to another timecode that conforms to the second timing system, or to convert a particular timecode that conforms to the second timing system to another timecode that conforms to the first timing system, and to allow at least one component of the first content and at least one component of the second content to be rendered based on a common timeline and presented in a synchronized fashion.

26. A method for synchronizing multiple contents, comprising:

receiving a plurality of contents including a first and a second content;

performing watermark extraction operations on the first content to recover a first timecode that is incorporated as part of a watermark payload into a first segment of the first content, the first timecode conforming to a first type of timing system;

determining a timing system associated with the second content;

upon a determination that the timing system associated with the second content is a second timing system that is different from the first timing system, obtaining a mapping that allows one or both of;

a conversion of timecodes from the first timing system to the second timing system or a conversion of timecodes from the second timing system to the first timing system, wherein the mapping is obtained from an external database that is identified based on at least a portion of the extracted watermark payload; and

synchronizing presentation of the second content with the first content using the mapping, wherein the first timing system is a fixed timing system that is based on a reference timeline that begins from an initially embedded watermarked segment of the first content and the second timing systems provides timing information according to a timecode format selected from one of;

a coordinated universal time (UTC),

a linear timecode (LTC),

a vertical interval timecode (VITC),

a precision time protocol (PTP),

a network time protocol (NTP), or

a global positioning system (GPS); and

wherein;

the mapping allows conversion between the first timing system and any of the second timing systems, and

the synchronizing comprises using the mapping to convert a particular timecode that conforms to the first timing system to another timecode that conforms to the second timing system, or to convert a particular timecode that conforms to the second timing system to another timecode that conforms to the first timing system, and to allow at least one component of the first content and at least one component of the second content to be rendered based on a common timeline and presented in a synchronized fashion.

27. A device, comprising:

a processor, and

a memory including processor executable code, the processor executable code, when executed by the processor configures the device to;

receive a plurality of contents including a first and a second content;

perform watermark extraction operations on the first content to recover a first timecode that is incorporated as part of a watermark payload into a first segment of the first content, the first timecode conforming to a first type of timing system;

determine a timing system associated with the second content;

upon a determination that the timing system associated with the second content is a second timing system that is different from the first timing system, obtain a mapping that allows one or both of;

a conversion of timecodes from the first timing system to the second timing system or a conversion of timecodes from the second timing system to the first timing system, wherein the mapping is obtained from an external database that is identified based on at least a portion of the extracted watermark payload; and

synchronize presentation of the second content with the first content using the mapping, wherein the first timing system is a fixed timing system that is based on a reference timeline that begins from an initially embedded watermarked segment of the first content and the second timing systems provides timing information according to a timecode format selected from one of;

a coordinated universal time (UTC),

a linear timecode (LTC),

a vertical interval timecode (VITC),

a precision time protocol (PTP),

a network time protocol (NTP), or

a global positioning system (GPS); and

wherein;

the mapping allows conversion between the first timing system and any of the second timing systems, and

the synchronized presentation comprises using the mapping to convert a particular timecode that conforms to the first timing system to another timecode that conforms to the second timing system, or to convert a particular timecode that conforms to the second timing system to another timecode that conforms to the first timing system, and to allow at least one component of the first content and at least one component of the second content to be rendered based on a common timeline and presented in a synchronized fashion.

28. A computer program product, embodied on one or more non-tangible computer readable media, comprising:

program code for receiving a plurality of contents including a first and a second content;

program code for performing watermark extraction operations on the first content to recover a first timecode that is incorporated as part of a watermark payload into a first segment of the first content, the first timecode conforming to a first type of timing system;

program code for determining a timing system associated with the second content;

program code for upon a determination that the timing system associated with the second content is a second timing system that is different from the first timing system, obtaining a mapping that allows one or both of;

a conversion of timecodes from the first timing system to the second timing system or a conversion of timecodes from the second timing system to the first timing system, wherein the mapping is obtained from an external database that is identified based on at least a portion of the extracted watermark payload; and

program code for synchronizing presentation of the second content with the first content using the mapping, the first timing system is a fixed timing system that is based on a reference timeline that begins from an initially embedded watermarked segment of the first content and the second timing systems provides timing information according to a timecode format selected from one of;

a coordinated universal time (UTC),

a linear timecode (LTC),

a vertical interval timecode (VITC),

a precision time protocol (PTP),

a network time protocol (NTP), or

a global positioning system (GPS); and

wherein;

the mapping allows conversion between the first timing system and any of the second timing systems, and

the synchronizing comprises using the mapping to convert a particular timecode that conforms to the first timing system to another timecode that conforms to the second timing system, or to convert a particular timecode that conforms to the second timing system to another timecode that conforms to the first timing system, and to allow at least one component of the first content and at least one component of the second content to be rendered based on a common timeline and presented in a synchronized fashion.