Method and apparatus for adaptive synchronization of digital video and audio playback in a multimedia playback system

• US 6,016,166 A
• Filed: 08/31/1998
• Issued: 01/18/2000
• Est. Priority Date: 08/31/1998
• Status: Expired due to Term

• Current Assignee: IDPA Holdings Incorporated (InterDigital, Inc.)
• Original Assignee: Lucent Technologies, Inc. (Nokia Corporation)
• Inventors: Huang, Yan, Jones, Raymond K.
• Primary Examiner(s): Kostak, Victor R.
• Application Number:
  US09/144,161
• Time in Patent Office: 505 Days
• Field of Search: 348/512, 348/525, 348/722, 348/423, 386/96, 386/110, 370/350, 370/503, 370/509, 370/516, 345/327, 345/328
• US Class Current: 348/515
• CPC Class Codes: H04N 21/4305 Synchronising client clock from received content stream, e.g. locking decoder clock with encoder clock, extraction of the PCR packets arrangements for synchronising receiver with transmitter by comparing receiver clock with transmitter clock H04L7/0012; arrangements for synchronising receiver with transmitter wherein the receiver takes measures against momentary loss of synchronisation H04L7/0083
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/4341 Demultiplexing of audio and video streams

Abstract

Synchronization among digital video and digital audio signals is maintained in a multimedia player with limited or highly varying processing power. In an exemplary embodiment of the invention, a processor with stored program memory controls the decoding and playback of audio blocks and video frames, each stream independently stored in an order of intended playback time. The processor gives priority to the decoding and playback of audio blocks, and periodically determines whether a backlog of encoded video frames exists. Under video backlog conditions, the processor reduces the backlog by decoding only those backlogged video frames that must be decoded in order to decode and play the frame in the sequence of backlogged video frames with the latest intended playback time. After decoding this latest backlogged frame, the processor directs it to be played. Normal video decoding and playback then resumes for video frames following this latest backlogged video frame in sequence and in intended playback time.

Claims

1. A method, for use in a multimedia playback system that decodes and renders a combined sequence of encoded audio and video information, for reducing playback disruptions, said method comprising the steps of:

granting priority to decoding and rendering said audio information;

monitoring the rendering of decoded audio and video information to determine whether it is backlogged relative to the rendering of said audio information; and

when said video information is backlogged, reducing said video backlog by

decoding only the backlogged video information that must be decoded to render the most current video image in the sequence of said backlogged video information,

rendering only said most current video image from said backlog, and

discarding all other video information not necessary to produce said most current video image.

2. A method, for use in a multimedia playback system, for rendering a sequence of encoded video frames and audio blocks, said method comprising the steps of:

decoding a selectable number of said audio blocks from said audio and video sequence;

monitoring the rendering by said system of each audio block;

determining a number representing the number of backlogged video frames stored by said system at a time immediately after the rendering of each audio block; and

when said number of backlogged video frames exceeds a specified number, eliminating said backlog by

determining whether said backlogged video frames include any key frames,

identifying a smallest set of said backlogged video frames that must be decoded in order to render a most current video frame in the sequence of said backlogged video frames,

decoding said smallest set of backlogged video frames, and

rendering said most current video frame in said backlogged sequence from said smallest set of backlogged video frames.

3. The method of claim 2, when said sequence of backlogged video frames includes at least one video key frame, wherein said smallest set of backlogged video frames includes a most current key frame in said sequence of backlogged video frames and all other backlogged video frames that follow said most current key frame in said sequence of backlogged frames.

4. The method of claim 2, when said sequence of backlogged video frames includes no video key frames, wherein said smallest set of backlogged video frames includes all backlogged video frames.

5. The method of claim 2, wherein said specified number of video frames is equal to one.

6. The method of claim 2, wherein said selectable number of audio blocks is an integer from two to four.

7. The method of claim 2, wherein said sequence of backlogged video frames includes video frames whose scheduled time for playback occurs no earlier than the scheduled playback time for a video frame that is the most recently rendered video frame, and no later than the scheduled playback time for an audio frame that is the next audio frame scheduled for rendering.

8. The method of claim 7, wherein said scheduled audio playback time is determined as a function of a sequential index number for each audio block, a number representing the number of encoded samples provided per audio block, and an audio sampling rate.

9. The method of claim 7, wherein said scheduled video playback time is determined as a function of a sequential index number for each video frame and a video frame rate.

10. A multimedia system for sequentially rendering digital video and audio information with high user-perceived fidelity, said system comprising:

an audio decoding and playback section;

a video decoding and playback section; and

a processor with stored program memory, where said processor supports video and audio playback by;

decoding a selectable number of audio blocks from said audio and video sequence;

monitoring the completed rendering by said system of each audio block;

determining the number of backlogged video frames stored by said system at the time immediately after the rendering of each audio block;

determining whether said backlogged video frames include any key frames; and

when said number of backlogged video frames exceeds a specified number, eliminating said backlog by

identifying a fewest number of said backlogged video frames that must be decoded in order to render a most current video frame in the sequence of said backlogged video frames,

decoding said fewest number of backlogged video frames, and

rendering said most current video frame in said backlogged sequence.

11. The multimedia system of claim 10, wherein said audio decoding and playback section includes a buffer for encoded audio blocks, a decoder, a buffer for decoded blocks and an audio player.

12. The multimedia system of claim 11, wherein said audio buffers are implemented as circular data buffers.

13. The multimedia system of claim 10, wherein said video decoding and playback section includes a buffer for encoded video frames, a decoder, a buffer for decoded frames and a video player.

14. The multimedia system of claim 13, wherein said video buffers are implemented as circular data buffers.

15. The multimedia system of claim 10, wherein said multimedia system is selected from the group consisting of personal computers, engineering workstations and digital signal processors with multimedia playback capabilities.