Audio-based synchronization to media

• US 8,718,805 B2
• Filed: 05/27/2010
• Issued: 05/06/2014
• Est. Priority Date: 05/27/2009
• Status: Expired due to Fees

• Current Assignee: Spot411 Technologies, Inc.
• Original Assignee: Spot411 Technologies, Inc.
• Inventors: Shah, Ajay, Glitsch, Hans M.
• Primary Examiner(s): Nguyen, Duc
• Assistant Examiner(s): ELJAIEK, ALEXANDER L
• Application Number:
  US12/789,377
• Publication Number:
  US 20100305729A1
• Time in Patent Office: 1,440 Days
• Field of Search: 348/194, 348/500, 348/510, 348/512, 348/515, 375/354, 375/359, 386/201, 386/219, 386239-262, 707/697, 707/698, 709/231, 709/232, 709/248, 713/375, 713/400, 713/500, 715/201, 715/203, 700/94, 725/18, 725/105, 725/141
• US Class Current: 700/94
• CPC Class Codes: G06F 16/433 using audio data
• CPC Class Codes: G06F 16/4393 Multimedia presentations, e.g. slide shows, multimedia albums
• CPC Class Codes: H04N 21/2407 Monitoring of transmitted content, e.g. distribution time, number of downloads arrangements for monitoring programmes for broadcast or for distribution combined with broadcast H04H20/14
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58

Abstract

An audio channel of a time-based media presentation provides a basis for synchronizing to the presentation across a variety of platforms independent of when and where the presentation is being viewed. By pre-processing the media into a series of non-unique hashes, and similarly processing an audio stream of the media captured at a client device, a comparison can be made that yields an accurate time offset within the presentation. The comparison may usefully be performed over a data network using a server that hosts data from the pre-processed media, and a variety of applications may be deployed on the client device based on the resulting synchronization.

Claims

1. A method comprising:

receiving an audio portion of a time-based media presentation with a microphone of a client device;

sampling the audio on the client device to obtain a sequence of digital samples of the audio portion;

processing the sequence of digital samples on the client device to provide a plurality of hashes, each one of the plurality of hashes including a plurality of bits, each one of the plurality of hashes providing a non-unique representation of a segment of the audio portion, and each one of the plurality of hashes having a known relative time offset to each other one of the plurality of hashes;

transmitting the plurality of hashes and a unique identifier for the time-based media presentation to a server, wherein the unique identifier explicitly identifies the time-based media presentation to the server;

receiving from the server a time offset indicative of a current time offset within the time-based media presentation; and

synchronizing an application on the client device to the time-based media presentation based upon the time offset.

2. The method of claim 1 wherein the client device includes at least one of a mobile device, a cellular phone, a laptop computer, a notebook computer, and a netbook.

3. The method of claim 1 wherein the time-based media presentation includes one or more of a movie, a sports event, and a television broadcast.

4. The method of claim 1 wherein adjacent ones of the plurality of hashes are calculated from overlapping windows of the sequence of digital samples.

5. The method of claim 1 wherein processing includes filtering the sequence of digital samples with a low pass filter to provide a filtered output and transforming the filtered output with a discrete Fourier transform.

6. The method of claim 1 wherein processing includes normalizing a magnitude of the sequence of digital samples.

7. The method of claim 1 further comprising, on the server,

determining an allowable bit error for the plurality of hashes;

identifying for each one of the plurality of hashes a set of candidate hashes with a number of bitwise variations from the one of the plurality of hashes no greater than the allowable bit error;

locating any candidate time offsets in the time-based media presentation corresponding to each set of candidate hashes for each one of the plurality of hashes;

updating scores for the candidate time offsets corresponding to each set of candidate hashes;

selecting one of the candidate time offsets having a best one of the scores as the time offset within the time-based media presentation; and

transmitting the time offset to the client device.

8. The method of claim 7 wherein determining the allowable bit error includes receiving the allowable bit error from the client device.

9. The method of claim 8 wherein each one of the plurality of hashes consists of thirty two bits and the allowable bit error is eight bits.

10. The method of claim 7 wherein identifying a set of candidate hashes includes providing a binary tree of all possible values for the hash and traversing the binary tree in a manner that excludes branches for binary values that exceed the allowable bit error for the hash.

11. The method of claim 7 further comprising conditionally transmitting the time offset only when the best one of the scores exceeds a predetermined threshold.

12. The method of claim 1 further comprising transmitting supplemental information to the server including a hash sequence number that identifies an order of the plurality of hashes relative to one another.

13. The method of claim 1 wherein synchronizing includes displaying an indicator on the client device that indicates a synchronization with the time-based media presentation.

14. The method of claim 1 further comprising rendering additional content on the client device under control of the application, the additional content synchronized to the time-based media presentation.

15. The method of claim 14 wherein rendering additional content includes rendering one or more of a supplemental video stream, contextual information, advertising, and interactive content.

16. A computer program product for audio-based synchronization, the computer program product comprising computer executable code embodied on a non-transitory computer readable medium that, when executing on one or more computing devices, performs the steps of:

receiving an audio portion of a time-based media presentation with a microphone of a client device;

sampling the audio on the client device to obtain a sequence of digital samples of the audio portion;

processing the sequence of digital samples on the client device to provide a plurality of hashes, each one of the plurality of hashes including a plurality of bits, each one of the plurality of hashes providing a non-unique representation of a segment of the audio portion, and each one of the plurality of hashes having a known relative time offset to each other one of the plurality of hashes;

transmitting the plurality of hashes and a unique identifier for the time-based media presentation to a server, wherein the unique identifier explicitly identifies the time-based media presentation to the server;

receiving from the server a time offset indicative of a current time offset within the time-based media presentation; and

synchronizing an application on the client device to the time-based media presentation based upon the time offset.

17. The computer program product of claim 16 wherein the client device includes at least one of a mobile device, a cellular phone, a laptop computer, a notebook computer, and a netbook.

18. The computer program product of claim 16 wherein the time-based media presentation includes one or more of a movie, a sports event, and a television broadcast.

19. The computer program product of claim 16 wherein adjacent ones of the plurality of hashes are calculated from overlapping windows of the sequence of digital samples.