Synchronized, variable-speed playback of digitally recorded audio and video

• US 5,583,652 A
• Filed: 04/28/1994
• Issued: 12/10/1996
• Est. Priority Date: 04/28/1994
• Status: Expired due to Term

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Ware, Malcolm S.
• Primary Examiner(s): Chin, Tommy P.
• Assistant Examiner(s): LEE, Y YOUNG
• Application Number:
  US08/233,902
• Time in Patent Office: 957 Days
• Field of Search: 358/312, 358/335, 358/341, 358/343, 358/342, 360/10.1, 360/19.1, 348/515, 348/472, 348/441
• US Class Current: 386/206
• CPC Class Codes: G10H 1/0033 Recording/reproducing or transmission of music for electrophonic musical instruments
• CPC Class Codes: G10H 1/368 displaying animated or moving pictures synchronized with the music or audio part
• CPC Class Codes: G10H 2240/066 MPEG audio-visual compression file formats, e.g. MPEG-4 for coding of audio-visual objects
• CPC Class Codes: G10H 2250/571 Waveform compression, adapted for music synthesisers, sound banks or wavetables
• CPC Class Codes: G11B 20/00007 Time or data compression or expansion audio compression based on psychoacoustics G10L19/00; data processing for reproducing audio data at different playback speeds G10L21/04; video compression H04N19/00; data compression per se H03M7/30
• CPC Class Codes: G11B 20/10527 Audio or video recording; Data buffering arrangements G11B20/12 - G11B20/18 take precedence
• CPC Class Codes: G11B 20/14 using self-clocking codes
• CPC Class Codes: G11B 2020/10555 wherein the frequency, the amplitude, or other characteristics of the audio signal is taken into account
• CPC Class Codes: G11B 2020/10629 the buffer having a specific structure
• CPC Class Codes: G11B 2020/10638 First-in-first-out memories [FIFO] buffers
• CPC Class Codes: G11B 2020/10805 involving specific measures to prevent a buffer overflow
• CPC Class Codes: G11B 2020/10814 involving specific measures to prevent a buffer underrun
• CPC Class Codes: G11B 2220/2537 Optical discs
• CPC Class Codes: G11B 27/005 Reproducing at a different information rate from the information rate of recording for television signals H04N5/783
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/4341 Demultiplexing of audio and video streams
• CPC Class Codes: H04N 21/47217 for controlling playback functions for recorded or on-demand content, e.g. using progress bars, mode or play-point indicators or bookmarks specific graphical features in visual interfaces H04N21/4312
• CPC Class Codes: H04N 5/073 for mutually locking plural sources of synchronising signals, e.g. studios or relay stations
• CPC Class Codes: H04N 5/602 for digital sound signals
• CPC Class Codes: H04N 5/783 : Adaptations for reproducing at a rate different from the recording rate
• CPC Class Codes: H04N 5/85 : on discs or drums
• CPC Class Codes: H04N 5/937 : by assembling picture element blocks in an intermediate store
• CPC Class Codes: H04N 7/56 : Synchronising systems therefor
• CPC Class Codes: H04N 9/8042 : involving data reduction

Abstract

Method and system for providing user-controlled, continuous, synchronized variable-speed playback of a previously recorded digital audio/video presentation. The user directly controls the rate of playback and the audio and video remain synchronized. The audio is expanded or compressed using the time domain harmonic scaling method so that the pitch of the audio remains undistorted. Synchronization is maintained by allowing one clock to serve as the master time clock for the system. The clocks which can serve as the master time clock include the audio decoder clock, the video decoder clock, and the system clock. The invention is particularly useful in multimedia display systems designed to display MPEG data.

Claims

1. In a variable-speed, digital audio/video playback system, a method of maintaining synchronization of a stream of digital video data with a stream of digital audio data originally recorded at a specified rate in frames per second where an audio frame comprises samples recorded in a fixed period of time corresponding to an associated video frame recording rate in frames per second, the method comprising the steps of:

placing at least 2 frames of digital audio in an audio buffering means and at least 2 frames of digital video in a video buffering means;

reading a value for a user-selected audio scaling factor, C, where C is a proportion of the originally recorded specified rate;

decoding, scaling and playing, if the audio buffering means is not empty, an audio frame from the audio buffering means using the scaling factor C so that playback of the audio frame requires an audio playback time period inversely proportional to C;

decoding and displaying, if the video buffering means is not empty, a frame of video from the video buffering means which corresponds to the currently playing audio frame, for a time period equal to the current audio playback time period;

reading a frame of video from the video data stream into the video buffering means and a frame of audio from the audio data stream into the audio buffering means if the end of the data streams has not been reached; and

re-reading the value of the user-selected audio scaling factor C in preparation for scaling and playing the next audio frame from the audio buffering means if the user-selected value to be assigned to C has been changed since the start of the immediately preceding audio playback time period.

2. In a variable-speed, digital audio/video playback system having an audio buffering means for storing at least 2 frames of digital audio and a video buffering means for storing at least 2 frames of digital video, a method of maintaining synchronization of a stream of digital video data with a stream of digital audio data originally recorded at a specified rate in frames per second where an audio frame comprises samples recorded in a fixed period of time corresponding to an associated video frame recording rate in frames per second, the method comprising the steps of:

reading a value for a user-selected audio scaling factor, C, where C is a proportion of the originally recorded specified rate;

decoding, scaling and playing, if the audio buffering means is not empty, an audio frame from the audio buffering means using the scaling factor C so that playback of the audio frame requires an audio playback time period inversely proportional to C;

decoding and displaying, if the video buffering means is not empty, a frame of video from the video buffering means which corresponds to the currently playing audio frame, for a time period equal to the audio playback time period; and

reading a frame of video from the video data stream into the video buffering means and a frame of audio from the audio data stream into the audio buffering means if the end of the data streams has not been reached.

3. The method according to claim 2, further comprising the step of:

re-reading the value of the user-selected audio scaling factor C in preparation for scaling and playing another audio frame from the audio buffering means if the user-selected value to be assigned to C has been changed since the start of the immediately preceding audio playback time period.

4. A digital audio/video playback system for playing digitally encoded, simultaneous audio and video at variable speeds while maintaining synchronization between the audio and video, the system comprising:

means for receiving digital audio data and digital video data from a source;

video buffering means capable of holding at least two frames of video, the video buffering means connected to the receiving means;

audio buffering means capable of holding at least two frames of audio, the audio buffering means connected to the receiving means;

means for displaying a plurality of video frames in succession;

means for generating sounds from electronic signals; and

apparatus for maintaining synchronization of a stream of digital video data with a stream of digital audio data originally recorded at a specified rate in frames per second where each audio frame comprises samples recorded in a fixed period of time corresponding to an associated video frame recording rate in frames per second, the apparatus disposed between the audio and video buffering means and the generating and displaying means, the apparatus including;

means for reading a value for a user-selected audio scaling factor, C, where C is a proportion of the originally recorded specified rate;

means for decoding, scaling and playing an audio frame from the audio buffering means using the scaling factor C so that playback of the audio frame requires an audio playback time period inversely proportional to C; and

means for decoding and displaying a frame of video from the video buffering means which corresponds to the currently playing audio frame, for a time period equal to the audio playback time period.

5. Apparatus for maintaining synchronization during variable-speed playback of a stream of digital video data with a stream of digital audio data originally recorded at a specified rate in frames per second where an audio frame comprises samples recorded in a fixed period of time corresponding to an associated video frame recording rate in frames per second, the apparatus comprising:

means for receiving digital audio data and digital video data from a source;

video buffering means capable of holding at least two frames of video, the video buffering means connected to the receiving means;

audio buffering means capable of holding at least two frames of audio, the audio buffering means connected to the receiving means;

means for reading a value for a user-selected audio scaling factor, C, where C is proportion of the originally recorded specified rate;

means for decoding, scaling and playing an audio frame from the audio buffering means using the scaling factor C so that playback of the audio frame requires an audio playback time period inversely proportional to C; and

means for decoding and displaying a frame of video from the video buffering means which corresponds to the currently playing audio frame, for a time period equal to the audio playback time period.

6. A digital audio/video playback subsystem for playing digitally encoded, simultaneous audio and video at variable speeds while maintaining synchronization between the audio and video, the system comprising:

means for receiving digital audio data and digital video data from a source;

video buffering means capable of holding at least two frames of video, the video buffering means connected to the receiving means;

audio buffering means capable of holding at least two frames of audio, the audio buffering means connected to the receiving means;

means for connection to apparatus for displaying a plurality of video frames in succession;

means for connection to apparatus for generating sounds from electronic signals; and

apparatus for maintaining synchronization of a stream of digital video data with a stream of digital audio data originally recorded at a specified rate in frames per second where each audio frame comprises samples recorded in a fixed period of time corresponding to an associated video frame recording rate in frames per second, the apparatus disposed between the audio and video buffering means and the connection means, the apparatus including;

means for reading a value for a user-selected audio scaling factor, C, where C is a proportion of the originally recorded specified rate;

means for decoding, scaling and playing an audio frame from the audio buffering means using the scaling factor C so that playback of the audio frame requires an audio playback time period inversely proportional to C; and

means for decoding and displaying a frame of video from the video buffering means which corresponds to the currently playing audio frame, for a time period equal to the audio playback time period.

7. A system for providing synchronized display of video and audio data, a synchronized rate being set by a scaling factor input which may be varied, the system for being connected to a source of synchronized digital audio and video data, the system comprising:

means for buffering digital video data received from the source;

system clocking means, the system clocking means being controlled by the scaling factor input;

means, connected to the digital video buffering means, for outputting the digital video data to a video display, the rate of the outputting being controlled by the system clocking means;

means for buffering digital audio data from the source;

means, connected to the digital audio buffering means, for scaling and outputting the digital audio data to a sound generating means, the digital audio data outputting means controlling the scaling and the output rate of the digital audio data based upon the system clocking means so that the output of the scaled, digital audio data is in synchronism with the output of the digital video data;

means for connection to a sound generating means; and

means for connection to a video display.

8. The system according to claim 7 further comprising:

a sound generating means for playing digitally encoded sounds.

9. The system according to either of claims 7 or 8 further comprising:

a video display for displaying a plurality of frames of digital video data in succession.