Methods and Systems for Synchronizing Media

• US 20110276334A1
• Filed: 05/04/2011
• Published: 11/10/2011
• Est. Priority Date: 12/12/2000
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Shazam Entertainment Limited (Apple Inc.)
• Inventors: Mills, William Michael, Inghelbrecht, Philip Georges, Barton, Christopher Jacques Penrose, Mukherjee, Dheeraj Shankar, Powar, Rahul, Wang, Avery Li-Chun
• Granted Patent: US 8,996,380 B2
• Time in Patent Office: Days
• US Class Current: 704/270
• CPC Class Codes: G06F 16/95 Retrieval from the web
• CPC Class Codes: G10H 1/361 Recording/reproducing of accompaniment for use with an external source, e.g. karaoke systems
• CPC Class Codes: G10H 2220/011 Lyrics displays, e.g. for karaoke applications
• CPC Class Codes: G10L 19/00 Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis in musical instruments G10H
• CPC Class Codes: G10L 25/48 specially adapted for particular use

Abstract

Systems and methods of synchronizing media are provided. A client device may be used to capture a sample of a media stream being rendered by a media rendering source. The client device sends the sample to a position identification module to determine a time offset indicating a position in the media stream corresponding to the sampling time of the sample, and optionally a timescale ratio indicating a speed at which the media stream is being rendered by the media rendering source based on a reference speed of the media stream. The client device calculates a real-time offset using a present time, a timestamp of the media sample, the time offset, and optionally the timescale ratio. The client device then renders a second media stream at a position corresponding to the real-time offset to be in synchrony to the media stream being rendered by the media rendering source.

Claims

1. A method of synchronizing media, comprising:

a client device receiving a media sample of a media stream being rendered by a media rendering source, the media sample associated with a timestamp corresponding to a sampling time of the media sample;

determining a time offset indicating a time position in the media stream corresponding to the sampling time of the media sample;

determining a real-time offset using a real-time timestamp, the timestamp of the media sample, and the time offset, wherein the real-time timestamp indicates a present time; and

the client device rendering a second media stream at a position corresponding to the real-time offset to be in synchrony to the media stream being rendered by the media rendering source.

2. The method of claim 1, wherein the sampling time is a beginning time of the media sample.

3. The method of claim 1, further comprising selecting the second media stream based upon a content identification of the media sample.

4. The method of claim 1, wherein the step of determining a time offset is performed by the client device.

5. The method of claim 1, wherein the step of determining a time offset is performed by the client device:

sending the media sample to a position identification module; and

receiving from the position identification module the time offset.

6. The method of claim 1, wherein determining the real-time offset comprises:

T_r(t)=T_S+t−

T₀

wherein T_r(t) is the real-time offset, T_S is the time offset indicating the time position in the media stream corresponding to the sampling time of the media sample, t is the real-time timestamp, and T₀ is the timestamp of the media sample.

7. The method of claim 1, further comprising receiving the second media stream from a server.

8. The method of claim 1, further comprising receiving the second media stream from a local storage device of the client device.

9. The method of claim 1, wherein the media stream is a soundtrack, and wherein the second media stream is a movie.

10. The method of claim 1, wherein the media stream is a video, and wherein the second media stream is a soundtrack of the video.

11. The method of claim 1, wherein the second media stream is a higher quality version of the media stream.

12. The method of claim 1, wherein the second media stream is an alternative version of the media stream.

13. The method of claim 1, wherein the media stream is a song, and wherein the second media stream is guitar tablatures of the song.

14. The method of claim 1, wherein the media stream is a song, and wherein the second media stream is musical notes of the song.

15. The method of claim 1, wherein the media stream is a song, and wherein the second media stream is a music video of the song.

16. The method of claim 1, wherein the media stream is a song, and wherein the second media stream is textual lyrics of the song.

17. The method of claim 16, further comprising the client device animating the textual lyrics on a display in synchrony with the song being rendered by the media rendering source.

18. The method of claim 16, further comprising receiving a file including all lyrics of the song, wherein the lyrics are time-annotated lyrics.

19. The method of claim 16, further comprising receiving streaming lyrics to be synchronously displayed to the song being rendered by the media rendering source.

20. The method of claim 1, further comprising verifying that the rendering of the second media stream is in synchrony to the media stream being rendered by the media rendering source.

21. The method of claim 20, wherein verifying that the rendering of the second media stream is in synchrony to the media stream being rendered by the media rendering source comprises:

the client device sending media samples of the media stream at periodic intervals to a discontinuity detection module; and

the client device receiving from the discontinuity detection module an instruction to perform a resynchronization of the second media stream with the media stream being rendered by the media rendering sources.

22. The method of claim 20, wherein verifying that the rendering of the second media stream is in synchrony to the media stream being rendered by the media rendering source comprises:

the client device sending media samples of the media stream at periodic intervals to a discontinuity detection module; and

the client device receiving from the discontinuity detection module an instruction to stop rendering the second media stream.

23. The method of claim 20, wherein verifying that the rendering of the second media stream is in synchrony to the media stream being rendered by the media rendering source comprises:

the client device sending media samples of the media stream at periodic intervals to a discontinuity detection module; and

the client device receiving from the discontinuity detection module an instruction to perform a content identification of the media stream.

24. A non-transitory computer readable medium having stored therein instructions executable by a computing device to cause the computing device to perform functions of:

receiving a media sample of a media stream being rendered by a media rendering source, the media sample associated with a timestamp corresponding to a sampling time of the media sample;

determining a time offset indicating a time position in the media stream corresponding to the sampling time of the media sample;

determining a real-time offset using a real-time timestamp, the timestamp of the media sample, and the time offset, wherein the real-time timestamp indicates a present time; and

rendering a second media stream at a position corresponding to the real-time offset to be in synchrony to the media stream being rendered by the media rendering source.

25. The non-transitory computer readable medium of claim 24, wherein the instructions are executable by the computing device to perform functions of:

sending the media sample to a position identification module; and

receiving from the position identification module the time offset.

26. The non-transitory computer readable medium of claim 24, wherein determining the real-time offset comprises:

T_r(t)=T_S+t−

T₀

wherein T_r(t) is the real-time offset, T_S is the time offset indicating the time position in the media stream corresponding to the sampling time of the media sample, t is the real-time timestamp, and T₀ is the timestamp of the media sample.

27. The non-transitory computer readable medium of claim 24, wherein the media stream is a song, and wherein the second media stream is textual lyrics of the song.

28. The non-transitory computer readable medium of claim 24, wherein the instructions are executable by the computing device to perform functions of:

verifying that the rendering of the second media stream is in synchrony to the media stream being rendered by the media rendering source.

29. A client device comprising:

a media receiver configured to receive a media sample of a media stream being rendered by a media rendering source, wherein the media sample is associated with a timestamp corresponding to a sampling time of the media sample;

a processor configured to receive a time offset indicating a time position in the media stream corresponding to the sampling time of the media sample;

the processor configured to determine a real-time offset using a real-time timestamp, the timestamp of the media sample, and the time offset, wherein the real-time timestamp indicates a present time; and

the client device configured to render a second media stream at a position corresponding to the real-time offset to be in synchrony to the media stream being rendered by the media rendering source.

30. The client device of claim 29, wherein the processor is configured to:

send the media sample to a position identification module; and

receive from the position identification module the time offset.

31. The client device of claim 29, wherein the processor is configured to determine the real-time offset by:

T_r(t)=T_S+t−

T₀

wherein T_r(t) is the real-time offset, T_S is the time offset indicating the time position in the media stream corresponding to the sampling time of the media sample, t is the real-time timestamp, and T₀ is the timestamp of the media sample.

32. A system comprising:

a server configured to receive a media sample of a media stream being rendered by a media rendering source, the media sample associated with a timestamp corresponding to a sampling time of the media sample, the server configured to determine a time offset indicating a time position in the media stream corresponding to the sampling time of the media sample; and

a client device configured to receive the time offset from the server and to determine a real-time offset using a real-time timestamp, the timestamp of the media sample, and the time offset, wherein the real-time timestamp indicates a present time, and the client device configured to render a second media stream at a position corresponding to the real-time offset to be in synchrony to the media stream being rendered by the media rendering source.

33. The system of claim 32, wherein the client device is configured to:

record the media sample;

send the media sample to the server; and

receive from the server the time offset.

34. The system of claim 32, wherein the client device is configured to determine the real-time offset by:

T_r(t)=T_S+t−

T₀

wherein T_r(t) is the real-time offset, T_S is the time offset indicating the time position in the media stream corresponding to the sampling time of the media sample, t is the real-time timestamp, and T₀ is the timestamp of the media sample.