Synchronizing text/visual information with audio playback

• US 7,058,889 B2
• Filed: 11/29/2001
• Issued: 06/06/2006
• Est. Priority Date: 03/23/2001
• Status: Expired due to Fees

• Current Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Original Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Inventors: Trovato, Karen I., Ramaswamy, Muralidharan, Li, Dongge
• Primary Examiner(s): Bayerl, Raymond J.
• Application Number:
  US09/998,033
• Publication Number:
  US 20020163533A1
• Time in Patent Office: 1,650 Days
• Field of Search: 715/716, 715/864, 715/700, 715/717, 715719-721, 715/723, 715/726, 715/727, 715/730, 715/731, 715/740, 715/762, 715/774, 715/500.1, 715/500, 715/512, 715/529, 715/526, 715/530, 715/531, 386/66, 386/85, 386/88, 386/95, 386/96, 360/19, 360/20, 360/24.01, 360/30.01, 360/30.04, 360/30.08, 360/30.2, 360/47.1, 360/47.11, 360/63, 360/65, 360/69, 360/70, 434/307.A, 434/307.R, 434/308, 434/318, 844/61, 844/62, 844/66, 84/3, 84/4, 844/53, 844/65, 369/19, 369/20, 369/24.01, 369/30.01, 369/30.04, 369/30.08, 369/30.2, 369/47.1, 369/47.11, 369/63, 369/65, 369/69, 369/70
• US Class Current: 715/716
• CPC Class Codes: G10H 1/368 displaying animated or moving pictures synchronized with the music or audio part
• CPC Class Codes: G10H 2210/041 based on mfcc [mel -frequency spectral coefficients]
• CPC Class Codes: G10H 2220/011 Lyrics displays, e.g. for karaoke applications
• CPC Class Codes: G10H 2240/325 Synchronizing two or more audio tracks or files according to musical features or musical timings synchronised lyrics, e.g. for karaoke G10H2220/011
• CPC Class Codes: G10H 2250/135 Autocorrelation
• CPC Class Codes: G10H 2250/235 Fourier transform; Discrete Fourier Transform [DFT]; Fast Fourier Transform [FFT]
• CPC Class Codes: G10L 25/48 specially adapted for particular use
• CPC Class Codes: H04N 21/2368 Multiplexing of audio and video streams
• CPC Class Codes: H04N 21/41265 having a remote control device for bidirectional communication between the remote control device and client device unidirectional remote control device H04N21/42204
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/43079 of additional data with content streams on multiple devices
• CPC Class Codes: H04N 21/4341 Demultiplexing of audio and video streams
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/44 Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
• CPC Class Codes: H04N 7/163 by receiver means only

Abstract

A method of synchronizing visual information with audio playback includes the steps of selecting a desired audio file from a list stored in memory associated with a display device, sending a signal from the display device to a separate playback device to cause the separate playback device to start playing the desired audio file; and displaying visual information associated with the desired audio file on the display device in accordance with timestamp data such that the visual information is displayed synchronously with the playing of the desired audio file, wherein the commencement of playing the desired audio file and the commencement of the displaying step are a function of the signal from the display device.

Claims

1. A method of synchronizing visual information with audio playback, comprising the steps of;

receiving a user selection of a desired audio file;

initiating play of the desired audio file on an audio playback device; and

displaying visual information associated with the desired audio file on a display device separate from said audio playback device in accordance with timestamp data such that the visual information is displayed synchronously with the playing of the desired audio file, wherein the commencement of playing the desired audio file by the audio playback device and the commencement of the displaying step by the display device are a function of a signal from the display device, said signal turning the audio playback device off and on prior to the audio playback device playing the desired audio file.

2. The method as claimed in claim 1, wherein the signal comprises an infrared signal.

3. The method as claimed in claim 1, wherein the display device comprises a handheld device.

4. The method as claimed in claim 1, wherein said method further comprises the step of:

analyzing the audio file and generating timestamp data corresponding to the visual information associated with the audio file.

5. The method as claimed in claim 4, wherein the timestamp data is generated utilizing tempo information extracted from music.

6. The method as claimed in claim 1, wherein the visual information comprises text lines.

7. The method as claimed in claim 1, wherein said method further comprises the step of:

displaying a title of the desired audio file on the display device.

8. The method as claimed in claim 1, wherein the selection is from a list stored in a memory associated with the display device.

9. The method as claimed in claim 1, wherein the selection is performed by a random number selection method.

10. The method as claimed in claim 1, wherein said method further comprises the step of:

sending a signal from the display device to the audio playback device to cause the audio playback device to start.

11. A method of synchronizing visual information with audio playback, comprising the steps of:

receiving a user selection of a desired audio file;

analyzing the audio file and generating timestamp data corresponding to the visual information associated with the audio file;

initiating play of the desired audio file on an audio playback device; and

displaying visual information associated with the desired audio file on a display device separate from said audio playback device in accordance with the timestamp data such that the visual information is displayed synchronously with the playing of the desired audio file, wherein the commencement of playing the desired audio file by the audio playback device and the commencement of the from the display device, and wherein the timestamp data is generated utilizing a text-based process.

12. A method of synchronizing visual information with audio playback, comprising the steps of:

receiving a user selection of a desired audio file;

analyzing the audio file and generating timestamp data corresponding to the visual information associated with the audio file;

initiating play of the desired audio file on an audio playback device; and

displaying visual information associated with the desired audio file on a display device separate from said audio playback device in accordance with the timestamp data such that the visual information is displayed synchronously with the playing of the desired audio file, wherein the commencement of playing the desired audio file by the audio playback device and the commencement of the displaying step by the display device are a function of a signal from the display device, and wherein the timestamp data is generated utilizing a pronunciation dictionary process.

13. A method of synchronizing visual information with audio playback, comprising the steps of:

receiving a user selection of a desired audio file;

analyzing the audio file and generating timestamp data corresponding to the visual information associated with the audio file;

initiating play of the desired audio file on an audio playback device; and

displaying visual information associated with the desired audio file on a display device separate from said audio playback device in accordance with the timestamp data such that the visual information is displayed synchronously with the playing of the desired audio file, wherein the commencement of playing the desired audio file by the audio playback device and the commencement of the displaying step by the display device are a function of a signal from the display device, and wherein the timestamp data is generated utilizing a note transcription of music process.

14. A method of synchronizing visual information with audio playback, comprising the steps of:

receiving a user selection of a desired audio file;

analyzing the audio file and generating timestamp data corresponding to the visual information associated with the audio file;

initiating play of the desired audio file on an audio playback device; and

displaying visual information associated with the desired audio file on a display device separate from said audio playback device in accordance with the timestamp data such that the visual information is displayed synchronously with the playing of the desired audio file, wherein the commencement of playing the desired is audio file by the audio playback device and the commencement of the displaying step by the display device are a function of a signal from the display device, and wherein said method further comprises the steps of;

comparing a location of a keyword extracted from the desired audio file with an actual location of a keyword within the timestamp data; and

adjusting the location of the extracted keyword to match the location of the keyword within the timestamp data.

15. An apparatus for synchronizing visual information associated with audio playback, comprising:

a memory for storing a listing of audio files, and for storing visual information corresponding to said audio files, said visual information containing timestamp data enabling synchronous display of said visual information with audio playback of the corresponding audio file on a separate audio playback device;

a display device for displaying said visual information; and

a processor for receiving a user selection of a desired audio file from said stored listing of audio files, for effecting playback of the desired audio file on said separate audio playback device, and for initiating display of the visual information associated with the desired audio file on the display device in accordance with the timestamp data, wherein the commencement of playing the desired audio file by a separate audio playback device and the commencement of the displaying step are a function of a signal from the display device, said signal turning the separate audio playback device off and on prior to the separate audio playback device playing the desired audio file.