Methods and systems for synchronizing media

• US 8,996,380 B2
• Filed: 05/04/2011
• Issued: 03/31/2015
• Est. Priority Date: 12/12/2000
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Shazam Entertainment Limited (Apple Inc.)
• Inventors: Wang, Avery Li-Chun, Powar, Rahul, Mills, William Michael, Inghelbrecht, Philip Georges, Mukherjee, Dheeraj Shankar, Barton, Christopher Jacques Penrose
• Primary Examiner(s): Desir, Pierre-Louis
• Assistant Examiner(s): Sirjani, Fariba
• Application Number:
  US13/100,825
• Publication Number:
  US 20110276334A1
• Time in Patent Office: 1,427 Days
• Field of Search: 704/270
• US Class Current: 704/270
• CPC Class Codes: G06F 16/95 Retrieval from the web
• CPC Class Codes: G10H 1/361 Recording/reproducing of accompaniment for use with an external source, e.g. karaoke systems
• CPC Class Codes: G10H 2220/011 Lyrics displays, e.g. for karaoke applications
• CPC Class Codes: G10L 19/00 Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis in musical instruments G10H
• CPC Class Codes: G10L 25/48 specially adapted for particular use

Abstract

Systems and methods of synchronizing media are provided. A client device may be used to capture a sample of a media stream being rendered by a media rendering source. The client device sends the sample to a position identification module to determine a time offset indicating a position in the media stream corresponding to the sampling time of the sample, and optionally a timescale ratio indicating a speed at which the media stream is being rendered by the media rendering source based on a reference speed of the media stream. The client device calculates a real-time offset using a present time, a timestamp of the media sample, the time offset, and optionally the timescale ratio. The client device then renders a second media stream at a position corresponding to the real-time offset to be in synchrony to the media stream being rendered by the media rendering source.

Claims

1. A method of synchronizing media, comprising:

a mobile device recording, from an ambient environment of the mobile device, a media sample of a media stream being rendered by a media rendering source, the media sample associated with a timestamp corresponding to a recording time of the media sample;

determining, based on a content identification of the media sample that results in an identified media track to which the media sample corresponds such that the identified media track is being rendered in the media stream by the media rendering source, a time offset indicating a time position within a length of the identified media track of the media sample corresponding to the recording time of the media sample;

determining a real-time offset using a real-time timestamp indicative of a present time, the timestamp of the media sample, and the time offset, wherein the real-time offset reflects an elapsed time since the recording time of the media sample; and

the mobile device rendering a second time-annotated media stream at a position in the time-annotated second media stream corresponding to the real-time offset to be in synchrony to the identified media track in the media stream being independently rendered in the ambient environment of the mobile device by the media rendering source.

2. The method of claim 1, wherein the recording time is a beginning time of the media sample.

3. The method of claim 1, further comprising selecting the second media stream based upon a content identification of the media sample.

4. The method of claim 1, wherein the step of determining a time offset is performed by the mobile device.

5. The method of claim 1, wherein the step of determining a time offset is performed by the mobile device:

sending the media sample to a position identification module; and

receiving from the position identification module the time offset.

6. The method of claim 1, wherein determining the real-time offset comprises:

T_r(t)=T_S+t−

T₀

wherein T_r(t) is the real-time offset, T_S is the time offset indicating the time position in the media stream corresponding to the recording time of the media sample, t is the real-time timestamp, and T₀ is the timestamp of the media sample.

7. The method of claim 1, further comprising receiving the second media stream from a server.

8. The method of claim 1, further comprising receiving the second media stream from a local storage device of the mobile device.

9. The method of claim 1, wherein the media stream is a soundtrack, and wherein the second media stream is a movie.

10. The method of claim 1, wherein the media stream is a video, and wherein the second media stream is a soundtrack of the video.

11. The method of claim 1, wherein the second media stream is a higher quality version of the media stream.

12. The method of claim 1, wherein the second media stream is an alternative version of the media stream.

13. The method of claim 1, wherein the media stream is a song, and wherein the second media stream is guitar tablatures of the song.

14. The method of claim 1, wherein the media stream is a song, and wherein the second media stream is musical notes of the song.

15. The method of claim 1, wherein the media stream is a song, and wherein the second media stream is a music video of the song.

16. The method of claim 1, wherein the media stream is a song, and wherein the second media stream is textual lyrics of the song.

17. The method of claim 16, further comprising the mobile device animating the textual lyrics on a display in synchrony with the song being rendered by the media rendering source.

18. The method of claim 16, further comprising receiving a file including all lyrics of the song, wherein the lyrics are time-annotated lyrics.

19. The method of claim 16, further comprising receiving streaming lyrics to be synchronously displayed to the song being rendered by the media rendering source.

20. The method of claim 1, further comprising verifying that the rendering of the second media stream is in synchrony to the media stream being rendered by the media rendering source.

21. The method of claim 20, wherein verifying that the rendering of the second media stream is in synchrony to the media stream being rendered by the media rendering source comprises:

the mobile device sending media samples of the media stream at periodic intervals to a discontinuity detection module; and

the mobile device receiving from the discontinuity detection module an instruction to perform a resynchronization of the second media stream with the media stream being rendered by the media rendering sources.

22. The method of claim 20, wherein verifying that the rendering of the second media stream is in synchrony to the media stream being rendered by the media rendering source comprises:

the mobile device sending media samples of the media stream at periodic intervals to a discontinuity detection module; and

the mobile device receiving from the discontinuity detection module an instruction to stop rendering the second media stream.

23. The method of claim 20, wherein verifying that the rendering of the second media stream is in synchrony to the media stream being rendered by the media rendering source comprises:

the mobile device sending media samples of the media stream at periodic intervals to a discontinuity detection module; and

the mobile device receiving from the discontinuity detection module an instruction to perform a content identification of the media stream.

24. A non-transitory computer readable medium having stored therein instructions executable by a computing device to cause the computing device to perform functions of:

recording, from an ambient environment of the computing device, a media sample of a media stream being rendered by a media rendering source, the media sample associated with a timestamp corresponding to a recording time of the media sample;

determining, based on a content identification of the media sample that results in an identified media track to which the media sample corresponds such that the identified media track is being rendered in the media stream by the media rendering source, a time offset indicating a time position within a length of the identified media track of the media sample corresponding to the recording time of the media sample;

determining a real-time offset using a real-time timestamp indicative of a present time, the timestamp of the media sample, and the time offset, wherein the real-time offset reflects an elapsed time since the recording time of the media sample; and

rendering a second time-annotated media stream at a position in the time-annotated second media stream corresponding to the real-time offset to be in synchrony to the identified media track in the media stream being independently rendered in the ambient environment of the client device by the media rendering source.

25. The non-transitory computer readable medium of claim 24, wherein the instructions are executable by the computing device to perform functions of:

sending the media sample to a position identification module; and

receiving from the position identification module the time offset.

26. The non-transitory computer readable medium of claim 24, wherein determining the real-time offset comprises:

T_r(t)=T_S+t−

T₀

wherein T_r(t) is the real-time offset, T_S is the time offset indicating the time position in the media stream corresponding to the recording time of the media sample, t is the real-time timestamp, and T₀ is the timestamp of the media sample.

27. The non-transitory computer readable medium of claim 24, wherein the media stream is a song, and wherein the second media stream is textual lyrics of the song.

28. The non-transitory computer readable medium of claim 24, wherein the instructions are executable by the computing device to perform functions of:

verifying that the rendering of the second media stream is in synchrony to the media stream being rendered by the media rendering source.

29. A mobile device comprising:

a media receiver configured to record, from an ambient environment of the mobile device, a media sample of a media stream being rendered by a media rendering source, wherein the media sample is associated with a timestamp corresponding to a recording time of the media sample;

a processor configured to receive, based on a content identification of the media sample that results in an identified media track to which the media sample corresponds such that the identified media track is being rendered in the media stream by the media rendering source, a time offset indicating a time position within a length of the identified media track of the media sample corresponding to the recording time of the media sample;

the processor configured to determine a real-time offset using a real-time timestamp indicative of a present time, the timestamp of the media sample, and the time offset, wherein the real-time offset reflects an elapsed time since the recording time of the media sample; and

the mobile device configured to render a second time-annotated media stream at a position in the second time-annotated media stream corresponding to the real-time offset to be in synchrony to the identified media track in the media stream being independently rendered in the ambient environment of the mobile device by the media rendering source.

30. The mobile device of claim 29, wherein the processor is configured to:

send the media sample to a position identification module; and

receive from the position identification module the time offset.

31. The mobile device of claim 29, wherein the processor is configured to determine the real-time offset by:

T_r(t)=T_S+t−

T₀

wherein T_r(t) is the real-time offset, T_S is the time offset indicating the time position in the media stream corresponding to the recording time of the media sample, t is the real-time timestamp, and T₀ is the timestamp of the media sample.

32. A system comprising:

a server configured to receive a recording, from an ambient environment of a mobile device, of a media sample of a media stream being rendered by a media rendering source, the media sample associated with a timestamp corresponding to a recording time of the media sample, the server configured to determine, based on a content identification of the media sample that results in an identified media track to which the media sample corresponds such that the identified media track is being rendered in the media stream by the media rendering source, a time offset indicating a time position within a length of the identified media track of the media sample corresponding to the recording time of the media sample; and

the mobile device configured to receive the time offset from the server and to determine a real-time offset using a real-time timestamp indicative of a present time, the timestamp of the media sample, and the time offset, wherein the real-time offset reflects an elapsed time since the recording time of the media sample, and the mobile device configured to render a second time-annotated media stream at a position in the second time-annotated media stream corresponding to the real-time offset to be in synchrony to the identified media track in the media stream being independently rendered in the ambient environment of the mobile device by the media rendering source.

33. The system of claim 32, wherein the mobile device is configured to:

record the media sample;

send the media sample to the server; and

receive from the server the time offset.

34. The system of claim 32, wherein the mobile device is configured to determine the real-time offset by:

T_r(t)=T_S+t−

T₀

wherein T_r(t) is the real-time offset, T_S is the time offset indicating the time position in the media stream corresponding to the recording time of the media sample, t is the real-time timestamp, and T₀ is the timestamp of the media sample.