Synchronizing ancillary data to content including audio

• US 10,715,879 B2
• Filed: 11/07/2016
• Issued: 07/14/2020
• Est. Priority Date: 04/08/2016
• Status: Active Grant

• Current Assignee: Source Digital, Inc.
• Original Assignee: Source Digital, Inc.
• Inventors: Carroll, Timothy J., Frecon, III, Henry Lawrence, Greene, Patrick Clay, Phillips, Michael Eric
• Primary Examiner(s): Bantamoi, Anthony
• Application Number:
  US15/344,715
• Publication Number:
  US 20170295394A1
• Time in Patent Office: 1,345 Days
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: H03G 7/002 in untuned or low-frequency amplifiers, e.g. audio amplifiers H03G7/007, H03G7/001, H03G7/008, H03G7/02, H03G7/06 take precedence
• CPC Class Codes: H03G 7/007 of digital or coded signals
• CPC Class Codes: H04N 21/231 Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
• CPC Class Codes: H04N 21/233 Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/234 Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
• CPC Class Codes: H04N 21/242 Synchronization processes, e.g. processing of PCR [Program Clock References] arrangements for synchronising broadcast or distribution via plural systems in broadcast distribution systems H04H20/18
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/43074 of additional data with content streams on the same device, e.g. of EPG data or interactive icon with a TV program
• CPC Class Codes: H04N 21/43079 of additional data with content streams on multiple devices
• CPC Class Codes: H04N 21/433 Content storage operation, e.g. storage operation in response to a pause request, caching operations
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4722 for requesting additional data associated with the content
• CPC Class Codes: H04N 21/812 involving advertisement data advertising per se G06Q30/02
• CPC Class Codes: H04N 21/845 Structuring of content, e.g. decomposing content into time segments
• CPC Class Codes: H04N 21/8586 by using a URL processing chained hypermedia data for information retrieval G06F16/94; information retrieval from the Internet by using URLs G06F16/955; URL in broadcast information H04H20/93; Web-based protocols H04L67/02

Abstract

Synchronizing ancillary data to content including audio includes obtaining a representation of the content'"'"'s audio and ancillary data pegged to instants in the representation of the content'"'"'s audio, and aligning the representation of the content'"'"'s audio to the content'"'"'s audio to synchronize the ancillary data pegged to the instants in the representation of the content'"'"'s audio to the content.

Claims

1. A machine or group of machines for synchronizing ancillary data to content including audio, comprising:

a pre-synchronizer configured to obtain a first ancillary data, a second ancillary data, a third ancillary data, and a representation of the content'"'"'s audio different from the content'"'"'s audio, and to time align the first, second, and third ancillary data to the representation of the content'"'"'s audio by time aligning the first ancillary data to a first portion of the representation of the content'"'"'s audio corresponding to a first relevant portion of the content and the second and third ancillary data to a second portion of the representation of the content'"'"'s audio, different from the first portion of the representation of the content'"'"'s audio, corresponding to a second relevant portion of the content, such that subsequent alignment of the representation of the content'"'"'s audio to the content'"'"'s audio synchronizes the first, second, and third ancillary data to the first relevant portion and the second relevant portion, respectively, such that the first ancillary data appears coextensive with the first relevant portion of the content and the second and third ancillary data appear coextensive with the second relevant portion of the content; and

a transceiver configured to communicate the representation of the content'"'"'s audio and the ancillary data time aligned to the representation of the content'"'"'s audio to a storage location,

the pre-synchronizer is configured to collect the ancillary data during a previous playout, previous broadcast, or previous performance of the content and to time align the ancillary data to the representation of the content'"'"'s audio.

2. The machine or group of machines of claim 1, wherein the pre-synchronizer is configured to receive ancillary data in the form of:

data about music in the audio,

data about a script played out in the content,

data about wardrobe worn by characters in the content,

data describing the content,

data including comments from performers, producers, or directors of the content,

data including comments from viewers of the content,

data including a Uniform Resource Locator (URL) to a resource that includes information about the content, or

advertisement data.

3. The machine or group of machines of claim 1, wherein:

the transceiver is configured to communicate the representation of the content'"'"'s audio and the ancillary data to a database accessible by consumers of the content.

4. The machine or group of machines of claim 1, wherein:

the transceiver is configured to communicate the representation of the content'"'"'s audio and the ancillary data to a storage location accessible by consumers of the content; and

the pre-synchronizer is configured to create a link to the storage location.

5. The machine or group of machines of claim 1, comprising:

an audio processor configured to create the representation of the content'"'"'s audio, wherein;

the transceiver is configured to communicate the representation of the content'"'"'s audio and the ancillary data to a storage location accessible by consumers of the content;

the pre-synchronizer is configured to create a link to the storage location; and

the audio processor is configured to insert the link to the storage location into metadata of the content'"'"'s audio.

6. The machine or group of machines of claim 1, comprising:

an audio processor configured to create the representation of the content'"'"'s audio and transmit the representation of the content'"'"'s audio to the pre-synchronizer.

7. The machine or group of machines of claim 1, comprising:

an audio processor configured to create the representation of the content'"'"'s audio by creating an audio waveform signature of the content'"'"'s audio.

8. The machine or group of machines of claim 1, comprising:

authoring tools configured to collect the ancillary data during a playout or performance of the content for time aligning the ancillary data to the representation of the content'"'"'s audio.

9. A machine or group of machines for synchronizing ancillary data to content including audio, comprising:

a transceiver configured to, upon a comparison of the content'"'"'s audio to a representation of the content'"'"'s audio different from the content'"'"'s audio to identify the content, obtain from a remote storage location first, second, and third ancillary data time aligned to the representation of the content'"'"'s audio, wherein the first ancillary data is time aligned to a first portion of the representation of the content'"'"'s audio corresponding to a first relevant portion of the content and the second and third ancillary data are time aligned to a second portion of the representation of the content'"'"'s audio, different from the first portion of the representation of the content'"'"'s audio, corresponding to a second relevant portion of the content;

a post-synchronizer configured to align the representation of the content'"'"'s audio to the content'"'"'s audio thereby synchronizing the first, second, and third ancillary data to the first relevant portion and the second relevant portion, respectively, such that the first ancillary data appears coextensive with the first relevant portion of the content and the second and third ancillary data appear coextensive with the second relevant portion of the content; and

interaction tools configured to collect the ancillary data during a playout or performance of the content for time aligning the ancillary data to the representation of the content'"'"'s audio.

10. The machine or group of machines of claim 9, wherein the transceiver is configured to receive the ancillary data in the form of:

data about music in the audio,

data about a script played out in the content,

data about wardrobe worn by characters in the content,

data describing the content,

data including comments from performers, producers, or directors of the content,

data including comments from viewers of the content,

data including a Uniform Resource Locator (URL) to a resource that includes information about the content, or

advertisement data.

11. The machine or group of machines of claim 9, comprising:

an audio processor configured to receive the content'"'"'s audio and extract from metadata of the content'"'"'s audio a link to a storage location in which the representation of the content'"'"'s audio and the ancillary data are stored and accessible by consumers of the content; and

the transceiver is configured to obtain the representation of the content'"'"'s audio and the ancillary data from the storage location.

12. The machine or group of machines of claim 9, comprising:

interaction tools configured to display the ancillary data in synchronicity with presentation of the content relying on the aligning of the representation of the content'"'"'s audio to the content'"'"'s audio to synchronize the ancillary data to the content'"'"'s audio.

13. A machine or group of machines for synchronizing ancillary data to content including audio, comprising:

a database configured to store a representation of the content'"'"'s audio different from the content'"'"'s audio, a first ancillary data time aligned to a first portion of the representation of the content'"'"'s audio corresponding to a first relevant portion of the content, and a second and a third ancillary data time aligned to a second portion of the representation of the content'"'"'s audio, different from the first portion of the representation of the content'"'"'s audio, corresponding to a second relevant portion of the content such that subsequent alignment of the representation of the content'"'"'s audio to the content'"'"'s audio synchronizes the first, second, and third ancillary data to the first relevant portion and the second relevant portion, respectively, such that the first ancillary data appears coextensive with the first relevant portion of the content and the second and third ancillary data appear coextensive with the second relevant portion of the content; and

a transceiver configured to, upon a comparison of a) the content'"'"'s audio as received by a remote device to b) the representation of the content'"'"'s audio to identify the content, communicate at least some of the ancillary data to the remote device, wherein

the database is configured to continue to receive and store the ancillary data during subsequent playout or performance of the content, wherein the ancillary data is time aligned to the representation of the content'"'"'s audio during the subsequent playout or performance.

14. A method to synchronize ancillary data to content including audio, the method comprising:

obtaining a representation of the content'"'"'s audio different from the content'"'"'s audio and ancillary data time aligned to the representation of the content'"'"'s audio by correlation, wherein a first ancillary data is time aligned to a first portion of the representation of the content'"'"'s audio corresponding to a first relevant portion of the content, and a second and a third ancillary data is time aligned to a second portion of the representation of the content'"'"'s audio, different from the first portion of the representation of the content'"'"'s audio, corresponding to a second relevant portion of the content different from the first relevant portion;

upon a comparison of a) the content'"'"'s audio as received by a remote device to b) the representation of the content'"'"'s audio to identify the content, communicating the representation of the content'"'"'s audio and the ancillary data time aligned to the representation of the content'"'"'s audio such that subsequent alignment of the representation of the content'"'"'s audio to the content'"'"'s audio synchronizes the first, second, and third ancillary data to the first relevant portion and the second relevant portion, respectively, such that the first ancillary data appears coextensive with the first relevant portion of the content and the second and third ancillary data appear coextensive with the second relevant portion of the content; and

collecting the ancillary data during a playout or performance of the content, wherein the representation of the content'"'"'s audio is derived during the playout or performance from the content'"'"'s audio and wherein the ancillary data is time aligned to the representation of the content'"'"'s audio during the playout or performance.

15. The method of claim 14, comprising at least one of collecting, storing, retrieving or displaying the ancillary data, the ancillary data including:

data about music in the audio,

data about a script played out in the content,

data about wardrobe worn by characters in the content,

data describing the content,

data including comments from performers, producers, or directors of the content,

data including comments from viewers of the content,

data including a Uniform Resource Locator (URL) to a resource that includes information about the content, or

advertisement data.

16. The method of claim 14, comprising:

storing the representation of the content'"'"'s audio and the ancillary data to a database.

17. The method of claim 14, comprising:

creating the representation of the content'"'"'s audio by creating an audio waveform signature of the content'"'"'s audio; and

storing the representation of the content'"'"'s audio and the ancillary data.

18. The method of claim 14, comprising:

creating the representation of the content'"'"'s audio and time aligning the ancillary data to the content'"'"'s audio;

storing the representation of the content'"'"'s audio and the ancillary data to a storage location; and

creating a link to the storage location.

19. The method of claim 14, comprising:

storing the representation of the content'"'"'s audio and the ancillary data to a storage location;

creating a link to the storage location; and

inserting the link to the storage location into metadata of the content'"'"'s audio.

20. The method of claim 14, comprising:

displaying the ancillary data in synchronicity with presentation of the content relying on the aligning of the representation of the content'"'"'s audio to the content'"'"'s audio to synchronize the ancillary data to the content'"'"'s audio.