Video synchronization based on an audio cue

• US 9,392,144 B2
• Filed: 06/23/2014
• Issued: 07/12/2016
• Est. Priority Date: 06/23/2014
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Luthra, Gaurav, Biswas, Sanjeev Kumar, Chatterjee, Arijit, Munshi, Kausar
• Primary Examiner(s): Harold, Jefferey
• Assistant Examiner(s): KARWAN, SIHAR A
• Application Number:
  US14/312,173
• Publication Number:
  US 20150373231A1
• Time in Patent Office: 750 Days
• Field of Search: 725/151, 725/105
• US Class Current: 1/1
• CPC Class Codes: H04N 21/233 Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/242 Synchronization processes, e.g. processing of PCR [Program Clock References] arrangements for synchronising broadcast or distribution via plural systems in broadcast distribution systems H04H20/18
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/437 Interfacing the upstream path of the transmission network, e.g. for transmitting client requests to a VOD server flow control in data networks H04L47/10; streaming protocols, e.g. RTP or RTCP, H04L65/65; scheduling or organising the servicing of application requests in data packet switching networks H04L67/60
• CPC Class Codes: H04N 21/8113 comprising music, e.g. song in MP3 format
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56

Abstract

Techniques are disclosed for manipulating a media player based on the environment in which content is consumed. For example, a user listening to a radio broadcast or some other ambient sound hears a song begin to play. Recognizing the song, the user wishes to watch an associated music video. A smartphone is used to record a portion of the ambient sound using an application configured according to certain disclosed embodiments. The observed audio is compared with one or more archived audio segments, each of which is associated with corresponding video content. If a match is found between the observed audio segment and an archived audio segment, video content corresponding to the matched archived audio segment is played back via a media player installed on the device. The playback is synchronized with the ambient sound. This allows the user to enjoy both the ambient audio and corresponding video content.

Claims

1. A method for synchronizing observed audio with archived video content, the method comprising:

receiving an observed audio segment from a client computing device, wherein the observed audio segment corresponds to ambient audio recorded by the client computing device;

generating a plurality of hash values corresponding to the observed audio segment;

performing a comparison of each of the plurality of hash values to a plurality of archived hash values, wherein each of the plurality of archived hash values (a) is associated with one of a plurality of archived audio segments, and (b) is stored in a data repository that is separate from the client computing device;

identifying a selected archived audio segment and a time lag based on the comparison, wherein a portion of the selected archived audio segment corresponds to the observed audio segment;

identifying video content corresponding to the selected archived audio segment, wherein the identified video content is stored in a video content repository that is separate from the client computing device; and

streaming the identified video content to the client computing device from a time point based on the time lag.

2. The method of claim 1, wherein:

a plurality of time lags are identified based on the comparison; and

the method further comprises selecting one of the plurality of time lags based on receipt of an additional audio segment from the client computing device.

3. The method of claim 1, further comprising:

generating a synchronization map that includes a matching hash value that is found in both the plurality of hash values corresponding to the observed audio segment and the plurality of archived hash values, wherein;

the matching hash value is keyed to one or more (observed, archived) time pairs,

the observed time corresponds to a time of the observed audio segment at which the matching hash value was found, and

the archived time corresponds to a time of a potentially matching archived audio segment at which the matching hash value was found; and

generating a time lag data map that includes (a) a listing of one or more time lags derived from the synchronization map, wherein each of the one or more time lags is defined as a difference between the observed time and the archived time, and (b) a frequency count corresponding to each of the one or more time lags.

4. The method of claim 1, wherein:

each of the plurality of hash values corresponding to the observed audio segment is paired with a time of the observed audio segment at which the hash value was generated; and

each of the plurality of archived hash values is paired with a time of the associated archived audio segment at which the archived hash value was generated.

5. The method of claim 1, further comprising generating a synchronization map that includes a matching hash value that is found in both the plurality of hash values corresponding to the observed audio segment and the plurality of archived hash values, wherein:

the matching hash value is keyed to one or more (observed, archived) time pairs;

the observed time corresponds to a time of the observed audio segment at which the matching hash value was found; and

the archived time corresponds to a time of a potentially matching archived audio segment at which the matching hash value was found.

6. The method of claim 1, further comprising:

receiving the plurality of archived audio segments before receiving the observed audio segment form the client computing device; and

generating the plurality of archived hash values.

7. The method of claim 1, further comprising:

receiving a multimedia content item before receiving the observed audio

segment from the client computing device, wherein the multimedia content item includes one of the plurality of archived audio segments and corresponding video content;

generating the plurality of archived hash values; and

storing the corresponding video content in the video content repository.

8. The method of claim 1, wherein generating the plurality of hash values corresponding to the observed audio segment further comprises:

dividing a frequency spectrum of the observed audio segment into a plurality of frequency bands;

dividing each of the plurality of frequency bands into a plurality of bin subsets;

identifying a bin index corresponding to a maximum power in each of the plurality of bin subsets; and

generating a plurality of hash values over a duration of the observed audio segment based on the bin indices associated with each of the plurality of frequency bands, wherein each of the plurality of hash values are defined by a powered sum of the bin indices.

9. The method of claim 1, wherein generating the plurality of hash values corresponding to the observed audio segment further comprises:

dividing a frequency spectrum of the observed audio segment into a plurality of frequency bands;

dividing each of the plurality of frequency bands into a plurality of bin subsets;

identifying a bin index corresponding to a maximum power in each of the plurality of bin subsets; and

generating a plurality of hash values over a duration of the observed audio segment based on the bin indices associated with each of the plurality of frequency bands.

10. The method of claim 1, wherein generating the plurality of hash values corresponding to the observed audio segment further comprises:

dividing a frequency spectrum of the observed audio segment into a plurality of frequency bands;

dividing each of the plurality of frequency bands into a plurality of bin subsets;

identifying a bin index corresponding to a maximum power in each of the plurality of bin subsets; and

generating a plurality of hash values over a duration of the observed audio segment based on the bin indices associated with each of the plurality of frequency bands,

wherein the frequency spectrum is divided into 5, 6, 7, 8, 9 or 10 frequency bands, and each of the frequency bands is divided into 3, 4, 5, 6, 7 or 8 bin subsets.

11. A system for video synchronization that comprises a memory device and a processor that is operatively coupled to the memory device, wherein the processor is configured to execute instructions stored in the memory device that, when executed, cause the processor to carry out a process for synchronizing observed audio with archived video content, the process comprising:

receiving multimedia content that includes audio content and video content;

generating archived unique hash data based on the audio content;

storing the archived unique hash data in a data repository;

receiving an observed audio segment from a client computing device that is separate from the data repository;

generating observed unique hash data based on the observed audio segment;

storing in the memory device, a comprehensive time lag data map that correlates a plurality of archived audio segments with a list of (time lag, count) data pairs, wherein the time lag is based on a comparison of the archived unique hash data and the observed unique hash data, and wherein the count is based on a frequency of the paired time lag;

identifying a matching archived audio segment that corresponds to the observed audio segment based on a maximum count identified from the comprehensive time lag data map; and

,

transmitting video content that was received with the matching archived audio segment to the client computing device.

12. The system of claim 11, wherein the process for synchronizing observed audio with archived video content further comprises sending the client computing device an instruction to terminate transmission of the observed audio segment in response to receipt of a predetermined duration of the observed audio segment.

13. The system of claim 11, wherein the process for synchronizing observed audio with archived video content further comprises receiving a second observed audio segment from the client computing device in response to detecting that the maximum count identified from the comprehensive time lag data map is associated with a plurality of time lags.

14. The system of claim 11, wherein the video content that was received with the matching archived audio segment is streamed to the client computing device.

15. The system of claim 11, further comprising:

a video content repository configured to store the video content included in the received multimedia content, wherein;

the video content repository is separate from the client computing device;

and

the process for synchronizing observed audio with archived video content further comprises retrieving the video content from the video content repository.

16. The system of claim 11, wherein the process for synchronizing observed audio with archived video content further comprises recording the observed audio segment by the client computing.

17. A non-transitory computer readable medium with instructions that, when executed by one or more processors, causes a process for synchronizing observed audio with archived content to be carried out, the process comprising:

receiving an observed audio segment from a client computing device wherein the observed audio segment corresponds to ambient audio recorded by the client computing device;

identifying an archived audio segment that includes at least a portion of the observed audio segment, wherein the archived audio segment is stored in a data repository that is separate from the client computing device;

determining a time lag corresponding to a relative time offset between the observed audio segment and the archived audio segment;

transmitting archived content to the client computing device, wherein the archived content is associated with the archived audio segment, and wherein the archived content is transmitted from a time point that is at least partially based on the time lag.

18. The non-transitory computer readable medium of claim 17, wherein the archived content comprises visual content.

19. The non-transitory computer readable medium of claim 17, wherein the process further comprises receiving the archived audio segment before receiving the observed audio segment, wherein the archived audio segment is not received from the client computing device.

20. The non-transitory computer readable medium of claim 17, wherein identifying the archived audio segment further comprises:

generating a plurality of hash values corresponding to the observed audio segment; and

performing a comparison of each of the plurality of hash values to a plurality of archived hash values, wherein each of the plurality of archived hash values is associated with one of a plurality of archived audio segments.