Synchronizing ancillary data to content including audio

• US 10,397,663 B2
• Filed: 04/08/2016
• Issued: 08/27/2019
• Est. Priority Date: 04/08/2016
• Status: Active Grant

• Current Assignee: Source Digital, Inc.
• Original Assignee: Source Digital, Inc.
• Inventors: Carroll, Timothy J., Frecon, III, Henry Lawrence, Greene, Patrick Clay, Phillips, Michael Eric
• Primary Examiner(s): Idowu, Olugbenga O
• Application Number:
  US15/094,538
• Publication Number:
  US 20170295412A1
• Time in Patent Office: 1,236 Days
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: H03G 7/002 in untuned or low-frequency amplifiers, e.g. audio amplifiers H03G7/007, H03G7/001, H03G7/008, H03G7/02, H03G7/06 take precedence
• CPC Class Codes: H03G 7/007 of digital or coded signals
• CPC Class Codes: H04N 21/231 Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
• CPC Class Codes: H04N 21/233 Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/234 Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
• CPC Class Codes: H04N 21/242 Synchronization processes, e.g. processing of PCR [Program Clock References] arrangements for synchronising broadcast or distribution via plural systems in broadcast distribution systems H04H20/18
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/43074 of additional data with content streams on the same device, e.g. of EPG data or interactive icon with a TV program
• CPC Class Codes: H04N 21/43079 of additional data with content streams on multiple devices
• CPC Class Codes: H04N 21/433 Content storage operation, e.g. storage operation in response to a pause request, caching operations
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4722 for requesting additional data associated with the content
• CPC Class Codes: H04N 21/812 involving advertisement data advertising per se G06Q30/02
• CPC Class Codes: H04N 21/845 Structuring of content, e.g. decomposing content into time segments
• CPC Class Codes: H04N 21/8586 by using a URL processing chained hypermedia data for information retrieval G06F16/94; information retrieval from the Internet by using URLs G06F16/955; URL in broadcast information H04H20/93; Web-based protocols H04L67/02

Abstract

Synchronizing ancillary data to content including audio includes obtaining a representation of the content'"'"'s audio and ancillary data pegged to instants in the representation of the content'"'"'s audio, and aligning the representation of the content'"'"'s audio to the content'"'"'s audio to synchronize the ancillary data pegged to the instants in the representation of the content'"'"'s audio to the content.

Claims

1. A machine or group of machines for synchronizing ancillary data to content including audio, comprising:

a pre-synchronizer configured to obtain the ancillary data and a representation of the content'"'"'s audio and to peg the ancillary data to instants in the representation of the content'"'"'s audio corresponding to instants in the content such that subsequent alignment of the representation of the content'"'"'s audio to the content'"'"'s audio synchronizes the ancillary data, pegged to the instants in the representation of the content'"'"'s audio, to the content; and

a transceiver configured to communicate the representation of the content'"'"'s audio and the ancillary data pegged to the instants in the representation of the content'"'"'s audio to a storage location,

wherein the representation of the content'"'"'s audio is an audio waveform signature of the content'"'"'s audio.

2. The machine or group of machines of claim 1, wherein the pre-synchronizer is configured to receive ancillary data in the form of:

data about music in the audio,

data about a script played out in the content,

data about wardrobe wore by characters in the content, data describing the content,

data including comments from performers, producers, or directors of the content, data including comments from viewers of the content,

data including a Uniform Resource Locator (URL) to a resource that includes information about the content,

advertisement data, or

statistics of the content'"'"'s audio including at least one of;

content name or content identification, and

loudness or dynamic range scaling of the content'"'"'s audio.

3. The machine or group of machines of claim 1, wherein:

the transceiver is configured to communicate the representation of the content'"'"'s audio and the ancillary data pegged to the instants in the representation of the content'"'"'s audio to a database accessible by consumers of the content.

4. The machine or group of machines of claim 1, wherein:

the transceiver is configured to communicate the representation of the content'"'"'s audio and the ancillary data pegged to the instants in the representation of the content'"'"'s audio to a storage location accessible by consumers of the content; and

the pre-synchronizer is configured to create a link to the storage location.

5. The machine or group of machines of claim 1, comprising:

an audio processor configured to create the representation of the content'"'"'s audio,

wherein;

the transceiver is configured to communicate the representation of the content'"'"'s audio and the ancillary data pegged to the instants in the representation of the content'"'"'s audio to a storage location accessible by consumers of the content;

the pre-synchronizer is configured to create a link to the storage location; and

the audio processor is configured to insert the link to the storage location into metadata of the content'"'"'s audio.

6. The machine or group of machines of claim 1, comprising:

an audio processor configured to analyze the content'"'"'s audio to collect statistics of the content'"'"'s audio including at least one of loudness or dynamic range scaling of the content'"'"'s audio, wherein;

the transceiver is configured to communicate the representation of the content'"'"'s audio and the ancillary data pegged to the instants in the representation of the content'"'"'s audio to a storage location accessible by consumers of the content;

the pre-synchronizer is configured to create a link to the storage location; and

the audio processor is configured to insert the link to the storage location and the statistics of the content'"'"'s audio into metadata of the content'"'"'s audio.

7. The machine or group of machines of claim 1, comprising:

an audio processor configured to analyze the content'"'"'s audio to collect statistics of the content'"'"'s audio including at least one of loudness or dynamic range scaling of the content'"'"'s audio, wherein;

the transceiver is configured to communicate the representation of the content'"'"'s audio and the ancillary data pegged to the instants in the representation of the content'"'"'s audio to a storage location accessible by consumers of the content; and

the audio processor is configured to insert the statistics of the content'"'"'s audio into metadata of the content'"'"'s audio.

8. The machine or group of machines of claim 1, comprising:

an audio processor configured to analyze the content'"'"'s audio to collect statistics of the content'"'"'s audio including at least one of loudness or dynamic range scaling of the content'"'"'s audio, wherein;

the transceiver is configured to communicate the representation of the content'"'"'s audio, the statistics of the content'"'"'s audio and the ancillary data pegged to the instants in the representation of the content'"'"'s audio to a storage location accessible by consumers of the content.

9. The machine or group of machines of claim 1, wherein:

the pre-synchronizer is configured to collect the ancillary data during a previous playout or performance of the content and to peg the ancillary data to instants in the representation of the content'"'"'s audio corresponding to instants in the content'"'"'s audio.

10. The machine or group of machines of claim 1, comprising:

an audio processor configured to create the representation of the content'"'"'s audio and transmit the representation of the content'"'"'s audio to the pre-synchronizer.

11. The machine or group of machines of claim 1, comprising:

an audio processor configured to create the representation of the content'"'"'s audio by creating an audio waveform signature of the content'"'"'s audio.

12. The machine or group of machines of claim 1, comprising:

authoring tools configured to collect the ancillary data during a playout or performance of the content for pegging the ancillary data to the instants in the representation of the content'"'"'s audio corresponding to instants in the content'"'"'s audio.

13. A machine or group of machines for synchronizing ancillary data to content including audio, comprising:

a transceiver configured to obtain the content, a representation of the content'"'"'s audio that is an audio waveform signature of the content'"'"'s audio and ancillary data pegged to instants in the representation of the content'"'"'s audio corresponding to instants in the content; and

a post-synchronizer configured to align the representation of the content'"'"'s audio to the content'"'"'s audio thereby synchronizing the ancillary data pegged to the instants in the representation of the content'"'"'s audio to the content.

14. The machine or group of machines of claim 13, wherein the transceiver is configured to receive the ancillary data in the form of:

data about music in the audio,

data about a script played out in the content,

data about wardrobe wore by characters in the content, data describing the content,

data including comments from performers, producers, or directors of the content, data including comments from viewers of the content,

data including a Uniform Resource Locator (URL) to a resource that includes information about the content,

advertisement data, or

statistics of the content'"'"'s audio including at least one of;

content name or content identification, and

loudness or dynamic range scaling of the content'"'"'s audio.

15. The machine or group of machines of claim 13, comprising:

interaction tools configured to collect the ancillary data during a playout or performance of the content for pegging the ancillary data to the instants in the representation of the content'"'"'s audio corresponding to instants in the content'"'"'s audio.

16. The machine or group of machines of claim 13, comprising:

an audio processor configured to receive the content'"'"'s audio from the transceiver and extract from metadata of the content'"'"'s audio a link to storage location in which the representation of the content'"'"'s audio and the ancillary data pegged to the instants in the representation of the content'"'"'s audio are stored and accessible by consumers of the content; and

the transceiver is configured to obtain the representation of the content'"'"'s audio and the ancillary data pegged to the instants in the representation of the content'"'"'s audio from the storage location.

17. The machine or group of machines of claim 13, comprising:

an audio processor configured to receive the content'"'"'s audio from the transceiver and extract from metadata of the content'"'"'s audio a) a link to storage location in which the representation of the content'"'"'s audio and the ancillary data pegged to the instants in the representation of the content'"'"'s audio are stored accessible by consumers of the content and b) statistics of the content'"'"'s audio; and

the transceiver is configured to obtain the representation of the content'"'"'s audio and the ancillary data pegged to the instants in the representation of the content'"'"'s audio from the storage location.

18. The machine or group of machines of claim 13, comprising:

an audio processor configured to a) receive the content'"'"'s audio from the transceiver and extract from metadata of the content statistics of the content'"'"'s audio including at least one of loudness or dynamic range scaling of the content'"'"'s audio and b) process the content'"'"'s audio based on the statistics of the content'"'"'s audio.

19. The machine or group of machines of claim 13, comprising:

an audio processor configured to receive the content'"'"'s audio from the transceiver;

wherein

the transceiver is configured to obtain from a storage location the representation of the content'"'"'s audio, the ancillary data pegged to the instants in the representation of the content'"'"'s audio from a storage location accessible by consumers of the content, and statistics of the content'"'"'s audio including at least one of loudness or dynamic range scaling of the content'"'"'s audio, and

the audio processor is configured to process the content'"'"'s audio based on the statistics of the content'"'"'s audio.

20. The machine or group of machines of claim 13, comprising:

an audio processor configured to compare the content'"'"'s audio to the representation of the content'"'"'s audio to identify the content.

21. The machine or group of machines of claim 13, comprising:

an audio processor configured to,

when the content'"'"'s audio includes metadata, using statistics of the content'"'"'s audio including at least one of loudness or dynamic range scaling of the content'"'"'s audio stored in the metadata to process the content'"'"'s audio, and

when the content'"'"'s audio does not include metadata, using statistics of the content'"'"'s audio including at least one of loudness or dynamic range scaling of the content'"'"'s audio stored with the ancillary data to process the content'"'"'s audio.

22. The machine or group of machines of claim 13, comprising:

an audio processor configured to process the content'"'"'s audio using statistics of the content'"'"'s audio including at least one of loudness or dynamic range scaling of the content'"'"'s audio stored with the ancillary data.

23. The machine or group of machines of claim 13, comprising:

interaction tools configured to display the ancillary data in synchronicity with presentation of the content relying on the aligning of the representation of the content'"'"'s audio to the content'"'"'s audio to synchronize the ancillary data pegged to the instants in the representation of the content'"'"'s audio to the content'"'"'s audio.

24. A machine or group of machines for synchronizing ancillary data to content including audio, comprising:

a database configured to store a representation of the content'"'"'s audio that is an audio waveform signature of the content'"'"'s audio and ancillary data pegged to instants in the representation of the content'"'"'s audio such that subsequent alignment of the representation of the content'"'"'s audio to the content'"'"'s audio synchronizes the ancillary data pegged to the instants in the representation of the content'"'"'s audio to the content; and

a transceiver configured to communicate the representation of the content'"'"'s audio and the ancillary data pegged to the instants in the representation of the content'"'"'s audio.

25. The machine or group of machines of claim 24, wherein

the database is configured to store statistics of the content'"'"'s audio including at least one of loudness or dynamic range scaling of the content'"'"'s audio.

26. The machine or group of machines of claim 24, wherein

the database is configured to continue to receive and store the ancillary data during subsequent playout or performance of the content, wherein the ancillary data is pegged to the instants in the representation of the content'"'"'s audio corresponding to instants in the content'"'"'s audio during the subsequent playout or performance.

27. A method to synchronize ancillary data to content including audio, the method comprising:

obtaining a representation of the content'"'"'s audio that is an audio waveform signature of the content'"'"'s audio and ancillary data pegged to instants in the representation of the content'"'"'s audio; and

communicating the representation of the content'"'"'s audio and the ancillary data pegged to the instants in the representation of the content'"'"'s audio such that subsequent alignment of the representation of the content'"'"'s audio to the content'"'"'s audio synchronizes the ancillary data pegged to the instants in the representation of the content'"'"'s audio to the content.

28. The method of claim 27, comprising at least one of collecting, storing, retrieving or displaying the ancillary data, the ancillary data including:

data about music in the audio,

data about a script played out in the content,

data about wardrobe wore by characters in the content, data describing the content,

data including comments from performers, producers, or directors of the content, data including comments from viewers of the content,

data including a Uniform Resource Locator (URL) to a resource that includes information about the content,

advertisement data, or

statistics of the content'"'"'s audio including at least one of;

content name or content identification, and

loudness or dynamic range scaling of the content'"'"'s audio.

29. The method of claim 27, comprising:

pegging the ancillary data to the instants in the representation of the content'"'"'s audio; and

storing the representation of the content'"'"'s audio and the ancillary data pegged to the instants in the representation of the content'"'"'s audio to a database.

30. The method of claim 27, comprising:

collecting the ancillary data during a playout or performance of the content, wherein the representation of the content'"'"'s audio is derived during the playout or performance from the content'"'"'s audio and wherein the ancillary data is pegged to the instants in the representation of the content'"'"'s audio corresponding to instants in the content'"'"'s audio during the playout or performance.

31. The method of claim 27, comprising:

creating the representation of the content'"'"'s audio by creating an audio waveform signature of the content'"'"'s audio; and

storing the representation of the content'"'"'s audio and the ancillarydata pegged to the instants in the representation of the content'"'"'s audio.

32. The method of claim 27, comprising:

creating the representation of the content'"'"'s audio and pegging the ancillary data to the instants in the representation of the content'"'"'s audio;

storing the representation of the content'"'"'s audio and the ancillary data pegged to the instants in the representation of the content'"'"'s audio to a storage location; and

creating a link to the storage location.

33. The method of claim 27, comprising:

storing the representation of the content'"'"'s audio and the ancillary data pegged to the instants in the representation of the content'"'"'s audio to a storage location;

creating a link to the storage location; and

inserting the link to the storage location into metadata of the content'"'"'s audio.

34. The method of claim 27, comprising:

analyzing the content'"'"'s audio to collect statistics of the content'"'"'s audio including at least one of loudness or dynamic range scaling of the content'"'"'s audio;

storing the representation of the content'"'"'s audio and the ancillary data pegged to the instants in the representation of the content'"'"'s audio to a storage location;

creating a link to the storage location; and

inserting the link to the storage location and the statistics of the content'"'"'s audio into metadata of the content'"'"'s audio.

35. The method of claim 27, comprising:

comparing the content'"'"'s audio to the representation of the content'"'"'s audio to identify the content.

36. The method of claim 27, comprising:

analyzing the content'"'"'s audio to collect statistics of the content'"'"'s audio including at least one of loudness or dynamic range scaling of the content'"'"'s audio;

storing the representation of the content'"'"'s audio and the ancillary data pegged to the instants in the representation of the content'"'"'s audio to a storage location;

inserting the statistics of the content'"'"'s audio into metadata of the content'"'"'s audio.

37. The method of claim 27, comprising:

analyzing the content'"'"'s audio to collect statistics of the content'"'"'s audio including at least one of loudness or dynamic range scaling of the content'"'"'s audio; and

storing the representation of the content'"'"'s audio, the statistics of the content'"'"'s audio and the ancillary data pegged to the instants in the representation of the content'"'"'s audio to a storage location.

38. The method of claim 27, comprising:

when the content'"'"'s audio includes metadata, using statistics of the content'"'"'s audio stored in the metadata to bypass or adjust an audio processor processing the content'"'"'s audio, and

when the content'"'"'s audio does not include metadata, comparing the content'"'"'s audio to the representation of the content'"'"'s audio to identify the content and using statistics of the content'"'"'s audio stored with the ancillary data to bypass or adjust the audio processor processing the content'"'"'s audio.

39. The method of claim 27, comprising:

using statistics of the content'"'"'s audio stored with the ancillary data to bypass or adjust the audio processor processing the content'"'"'s audio.

40. The method of claim 27, comprising:

displaying the ancillary data in synchronicity with presentation of the content relying on the aligning of the representation of the content'"'"'s audio to the content'"'"'s audio to synchronize the ancillary data pegged to the instants in the representation of the content'"'"'s audio to the content'"'"'s audio.