Synchronized, variable-speed playback of digitally recorded audio and video

• US 5,664,044 A
• Filed: 03/27/1996
• Issued: 09/02/1997
• Est. Priority Date: 04/28/1994
• Status: Expired due to Term

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Ware, Malcolm S.
• Primary Examiner(s): Au, Amelia
• Assistant Examiner(s): LEE, Y YOUNG
• Application Number:
  US08/624,008
• Time in Patent Office: 524 Days
• Field of Search: 358/312, 358/335, 358/341, 358/343, 358/342, 360/10.1, 360/19.1, 348/515, 348/472, 348/441
• US Class Current: 386/206
• CPC Class Codes: G10H 1/0033 Recording/reproducing or transmission of music for electrophonic musical instruments
• CPC Class Codes: G10H 1/368 displaying animated or moving pictures synchronized with the music or audio part
• CPC Class Codes: G10H 2240/066 MPEG audio-visual compression file formats, e.g. MPEG-4 for coding of audio-visual objects
• CPC Class Codes: G10H 2250/571 Waveform compression, adapted for music synthesisers, sound banks or wavetables
• CPC Class Codes: G11B 20/00007 Time or data compression or expansion audio compression based on psychoacoustics G10L19/00; data processing for reproducing audio data at different playback speeds G10L21/04; video compression H04N19/00; data compression per se H03M7/30
• CPC Class Codes: G11B 20/10527 Audio or video recording; Data buffering arrangements G11B20/12 - G11B20/18 take precedence
• CPC Class Codes: G11B 20/14 using self-clocking codes
• CPC Class Codes: G11B 2020/10555 wherein the frequency, the amplitude, or other characteristics of the audio signal is taken into account
• CPC Class Codes: G11B 2020/10629 the buffer having a specific structure
• CPC Class Codes: G11B 2020/10638 First-in-first-out memories [FIFO] buffers
• CPC Class Codes: G11B 2020/10805 involving specific measures to prevent a buffer overflow
• CPC Class Codes: G11B 2020/10814 involving specific measures to prevent a buffer underrun
• CPC Class Codes: G11B 2220/2537 Optical discs
• CPC Class Codes: G11B 27/005 Reproducing at a different information rate from the information rate of recording for television signals H04N5/783
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/4341 Demultiplexing of audio and video streams
• CPC Class Codes: H04N 21/47217 for controlling playback functions for recorded or on-demand content, e.g. using progress bars, mode or play-point indicators or bookmarks specific graphical features in visual interfaces H04N21/4312
• CPC Class Codes: H04N 5/073 for mutually locking plural sources of synchronising signals, e.g. studios or relay stations
• CPC Class Codes: H04N 5/602 for digital sound signals
• CPC Class Codes: H04N 5/783 : Adaptations for reproducing at a rate different from the recording rate
• CPC Class Codes: H04N 5/85 : on discs or drums
• CPC Class Codes: H04N 5/937 : by assembling picture element blocks in an intermediate store
• CPC Class Codes: H04N 7/56 : Synchronising systems therefor
• CPC Class Codes: H04N 9/8042 : involving data reduction

Abstract

Method and system for providing user-controlled, continuous, synchronized variable-speed playback of a previously recorded digital audio/video presentation. The user directly controls the rate of playback and the audio and video remain synchronized. The audio is expanded or compressed using the time domain harmonic scaling method so that the pitch of the audio remains undistorted. Synchronization is maintained by allowing one clock to serve as the master time clock for the system. The clocks which can serve as the master time clock include the audio decoder clock, the video decoder clock, and the system clock. The invention is particularly useful in multimedia display systems designed to display MPEG data.

Claims

1. A method of maintaining synchronization of a stream of digital audio data with a stream of digital video data originally recorded at a specified rate in frames per second, FPS, the digital video data being played back on a variable-speed audio/video display system using a user-selected scaling factor, RV, which is set to a proportional value of a recorded rate, the method comprising the steps of:

placing at least 2 frames of digital audio in an audio buffer and at least 2 frames of digital video in a video buffer;

setting an audio time domain harmonic scaling factor, C, equal to the user-selected scaling factor, RV;

calculating an amount of time in seconds, T, required to play back an audio frame with the selected audio time domain harmonic scaling factor C;

decoding and displaying, if the video buffer is not empty, a frame of video from the video buffer for T seconds;

decoding, scaling and playing, if the audio buffer is not empty, an audio frame from the audio buffer using the audio time domain harmonic scaling factor C so that playback of the audio frame requires an audio playback time period equal to T seconds;

reading, after T seconds, and if the end of the stream of digital audio data or the stream of digital video data has not been reached, another frame from the stream of digital audio into the audio buffer and another frame from the stream of digital video into the video buffer;

determining whether the audio playback time period of the audio frame played during the preceding time period T was exactly equal to T seconds;

adjusting the value C if the audio playback time period was not exactly equal to T seconds; and

recalculating the time T if the audio time domain harmonic scaling factor C has changed since the beginning of the preceding time period T.

2. In a system for playing digitally recorded audio and video data streams, the system having an audio buffer holding two or more audio frames, and a video buffer holding two or more video frames, a method for maintaining synchronization during variable-speed playback of the audio data stream with the video data stream, where the video data stream is being played using a user-selected scaling factor, RV, which is set to a proportional value of an originally recorded rate, the method comprising the steps of:

calculating an amount of time in seconds, T, required to play back an audio frame with an audio time domain harmonic scaling factor C, where C is initially set to equal the user-selected scaling factor RV;

decoding and displaying, if the video buffer is not empty, a frame of video from the video buffer for T seconds;

decoding, scaling and playing, if the audio buffer is not empty, an audio frame from the audio buffer using the audio time domain harmonic scaling factor C so that playback of the audio frame requires an audio playback time period equal to T seconds; and

reading, after T seconds, and if the end of the data streams has not been reached, a frame from the stream of digital audio into the audio buffer and a frame from the stream of digital video into the video buffer.

3. The method according to claim 2 further comprising the steps of:

determining whether the audio buffer is approaching an underflow condition because the audio playback time period is less than T seconds; and

if the audio buffer is approaching an underflow condition, recalculating the audio time domain harmonic scaling factor C according to an equation C=C-A, where A is an audio adjustment factor which is set to a fractional value of C that is large enough to prevent underflow and overflow of the audio buffer, and small enough so that changes in playback speed are perceived as gradual by a user.

4. The method according to claim 3 further comprising the steps of:

determining whether the audio buffer is approaching an overflow condition because the audio playback time period is more than T seconds; and

if the audio buffer is approaching an overflow condition, recalculating the audio time domain harmonic scaling factor C according to an equation C=C+A.

5. The method according to claim 2 further comprising the steps of:

determining whether the audio buffer is in approaching an overflow condition because the audio playback time period is more than T seconds; and

if the audio buffer is approaching an overflow condition, recalculating the audio time domain harmonic scaling factor C according to an equation C=C+A, where A is an audio adjustment factor which is set to a fractional value of C that is large enough to prevent underflow and overflow of the audio buffer, but small enough so that changes in playback speed are perceived as gradual by a user.

6. A digital audio/video playback system for playing digitally encoded, simultaneous audio and video at variable speeds while maintaining synchronization between the audio and video, the system comprising:

means for receiving a digital audio data stream and digital video data stream from a source;

a video buffer capable of holding at least two frames of video, the video buffer connected to the receiving means;

an audio buffer capable of holding at least two frames of audio, the audio buffer connected to the receiving means;

means for displaying a plurality of video frames in succession;

means for generating sounds from electronic signals; and

apparatus for maintaining synchronization during playback of the digital audio data stream with the digital video data stream, where the digital video data stream is being played using a user-selected scaling factor, RV, which is set to a proportional value of an originally recorded rate, the apparatus being disposed between the audio and video buffers and the generating and displaying means, the apparatus including;

means for calculating an amount of time in seconds, T, required to play back an audio frame with an audio time domain harmonic scaling factor C, where C is initially set to equal RV;

a video decoder for decoding a frame of video from the video buffer and sending the frame of video to the display means; and

an audio time domain harmonic scaler for decoding, scaling and playing an audio frame from the audio buffer using the audio time domain harmonic scaling factor C so that playback of the audio frame requires an audio playback time period equal to T.

7. The system of claim 6 wherein the apparatus for maintaining synchronization during playback further includes:

means for determining whether the audio buffer is approaching an underflow condition because the audio playback time period is less than T seconds; and

means for recalculating the audio time domain harmonic scaling factor C according to an equation C=C-A, where A is an audio adjustment factor which is set to a fractional value of C that is large enough to prevent underflow and overflow of the audio and video buffers, but small enough so that changes in playback speed are perceived as gradual by a user.

8. The system of claim 7 wherein the apparatus further includes:

means for determining whether the audio buffer is approaching an overflow condition because the audio playback time period is more than T seconds; and

means for recalculating the audio time domain harmonic scaling factor C according to an equation C=C+A.

9. The system of claim 6 wherein the apparatus for maintaining synchronization during playback further includes:

means for determining whether the audio buffer is approaching an overflow condition because the audio playback time period is more than T seconds; and

means for recalculating the audio time domain harmonic scaling factor C according to an equation C=C+A, where A is an audio adjustment factor which is set to a fractional value of C that is large enough to prevent underflow and overflow of the audio buffer, but small enough so that changes in playback speed are perceived as gradual by a user.

10. Apparatus for maintaining synchronization during variable-speed playback of an audio data stream with a video data stream, where the video data stream is being played using a user-selected scaling factor, RV, which is set to a proportional value of an originally recorded rate, the apparatus comprising:

means for receiving digital audio data and digital video data from a source;

a video buffer capable of holding at least two frames of video, the video buffer connected to the receiving means;

an audio buffer capable of holding at least two frames of audio, the audio buffer connected to the receiving means;

means for calculating an amount of time in seconds, T, required to play back an audio frame with an audio time domain harmonic scaling factor C which is set equal to the user-selected scaling factor RV;

means for decoding and displaying a frame of video from the video buffer for T seconds; and

an audio time domain harmonic scaler for decoding, scaling and playing an audio frame from the audio buffer using the audio time domain harmonic scaling factor C so that playback of the audio frame requires an audio playback time period equal to T.

11. The apparatus of claim 10 further comprising:

means for determining whether the audio buffer is approaching an underflow condition because the audio playback time period is less than T seconds; and

means for recalculating the audio time domain harmonic scaling factor C according to an equation C=C-A, where A is an audio adjustment factor which is set to a fractional value of C that is large enough to prevent underflow and overflow of the audio buffer, but small enough so that changes in playback speed are perceived as gradual by a user.

12. The apparatus of claim 11 further comprising:

means for determining whether the audio buffer is approaching an overflow condition because the audio playback time period is more than T seconds; and

means for recalculating the audio time domain harmonic scaling factor C according to an equation C=C+A.

13. The apparatus of claim 10 further comprising:

means for determining whether the audio buffer is approaching an overflow condition because the audio playback time period is more than T seconds; and

means for recalculating the audio time domain harmonic scaling factor C according to an equation C=C+A, where A is an audio adjustment factor which is set to a fractional value of C that is large enough to prevent underflow and overflow of the audio buffer, but small enough so that changes in playback speed are perceived as gradual by a user.

14. A digital audio/video playback subsystem for playing digitally encoded, simultaneous audio and video at variable speeds while maintaining synchronization between the audio and video, the system comprising:

means for receiving a digital audio data stream and a digital video data stream from a source;

a video buffer capable of holding at least two frames of video, the video buffer connected to the receiving means;

an audio buffer capable of holding at least two frames of audio, the audio buffer connected to the receiving means;

means for connection to apparatus for displaying a plurality of video frames in succession;

means for connection to apparatus generating sounds from electronic signals; and

apparatus for maintaining synchronization during playback of the digital audio data stream with the digital video data stream, where the digital video data stream is being played using a user-selected scaling factor, RV, which is set to a proportional value of an originally recorded rate, the apparatus disposed between the audio and

video buffers and the connection means, the apparatus including;

means for calculating an amount of time in seconds, T, required to play back an audio frame with an audio time domain harmonic scaling factor C, where C is set equal to the user-selected scaling factor RV;

a video decoder for decoding a frame of video from the video buffer and sending the frame of video to the display apparatus connection means; and

an audio time domain harmonic scaler for scaling and playing an audio frame from the audio buffer using the audio time domain harmonic scaling factor C so that playback of the audio frame requires an audio playback time period equal to T.

15. The subsystem of claim 14 wherein the apparatus for maintaining synchronization during playback further includes:

means for determining whether the audio buffer is approaching an underflow condition because the audio playback time period is less than T seconds; and

means for recalculating the audio time domain harmonic scaling factor C according to an equation C=C-A, where A is an audio adjustment factor which is set to a fractional value of C that is large enough to prevent underflow and overflow of the audio buffer, but small enough so that changes in playback speed are perceived as gradual by a user.

16. The subsystem of claim 15 wherein the apparatus further includes:

means for determining whether the audio buffer is approaching an overflow condition because the audio playback time period is more than T seconds; and

means for recalculating the audio time domain harmonic scaling factor C according to an equation C=C+A.

17. The subsystem of claim 14 wherein the apparatus for maintaining synchronization during playback further includes:

means for determining whether the audio buffer is approaching an overflow condition because the audio playback time period is more than T seconds; and

means for recalculating the audio time domain harmonic scaling factor C according to an equation C=C+A, where A is an audio adjustment factor which is set to a fractional value of C that is large enough to prevent underflow and overflow of the audio buffer, but small enough so that changes in playback speed are perceived as gradual by a user.