Audio/video synchronization for application programs

• US 5,471,576 A
• Filed: 11/16/1992
• Issued: 11/28/1995
• Est. Priority Date: 11/16/1992
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Yee, Raymond L.
• Primary Examiner(s): Herndon, Heather R.
• Assistant Examiner(s): Burwell, Joseph R.
• Application Number:
  US07/976,628
• Time in Patent Office: 1,107 Days
• Field of Search: 395/154, 395/152, 360/14.1, 371/42, 340/825.14, 340/825.2, 348/512, 348/515, 381/124
• US Class Current: 715/203
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually

Abstract

A synchronization process in an application program records audio fields with video synchronization counts, and plays back the audio and video fields in synchronism by tracking the video fields against the video sync counts in the audio fields. The video sync counts correspond to the number of video field processed when the audio field is processed. During recording of audio and video fields for the multimedia presentation, the video fields are counted. The video field count is appended to and recorded with each audio field. During playback, the system compares the count of video fields displayed against the video field count appended to the audio field being presented. If the counts are different, the system either skips video fields, or repeats video fields to bring the video fields into synchronism with the audio fields.

Claims

1. In a computing system running a multi-threaded audio/visual application program, a synchronization method embedded in the application program for synchronizing an audio thread with a video thread in the application program, said audio and video threads processing respectively audio fields and video fields for playback through audio and video adapters attached to the computing system, said method comprising the computer implemented steps of:

storing the video fields, the audio fields, and a sync count with each audio field, each audio field sync count identifying the video field that the audio field is to be in synchronism with during playback;

in the audio thread, controlling the processing of each audio field stored in the system and sending a sync flag and the sync count as a sync value to the video thread;

in the video thread, controlling the processing of each video field stored in the system and counting the video fields as they are processed to provide an actual video field count;

in the video thread, comparing the sync value from the audio thread with the actual video field count and

said controlling step in the video thread synchronizing the playback of the audio fields and video fields by continuing the processing of video fields until the actual video field count matches the sync value.

2. The method of claim 1 wherein said storing step comprises the steps of:

starting the video thread and the audio thread so that the computing system is operating the audio thread and the video thread in parallel to store audio and video fields of an audio/video presentation;

in the video thread, getting each video field of the presentation and saving each video field in the computing system for subsequent playback in the audio/visual presentation;

in the video thread, counting the video fields as they are saved by said saving step and sending to the audio thread a sync count for each video field as the video field is saved; and

the audio thread, getting each audio field of the presentation and saving the audio field with the sync count of the video field that the audio field is to be in synchronism with during playback.

3. The method of claim 1 wherein said comparing step computes a difference between the sync count and the actual video field count.

4. The method of claim 3 wherein said synchronizing step by said controlling step in the video thread comprises the steps of:

getting the next video field for processing and playback if the difference is zero;

repeating a video field if the difference indicates the processed video field is played back prior to the processed audio field to be synchronized with the video field; and

skipping video fields if the difference indicates the processed video field is played back after the processed audio field to be synchronized with the video field.

5. In a computing system for generating audio/visual presentations, an apparatus for synchronizing the playback of video data fields with audio data fields, said computing system having a processor, storage devices, an audio adapter with an audio device for playing back audio fields, and a video adapter with a video device for playing back video fields, said apparatus being driven by an audio/visual computer program running on the computing system and said apparatus comprising:

first means for getting and saving a sequence of video fields in the computing system for subsequent playback in an audio/visual presentation;

second means for retrieving an audio field at the same time said first means is getting and saving video fields, said audio field to accompany the sequence of video fields during the playback of the audio/visual presentation;

third means for counting each video field, as the video field is stored by said second means and for sending a sync count to said second means as each video field is stored by said first means;

fourth means responsive to the sync count from said third means for appending the sync count to the audio field retrieved by said second means and for saving the audio field with the appended sync count in the computing system for subsequent playback with the sequence of audio fields in the audio/visual presentation;

audio sending means for sending a sequence of audio fields stored in the system to the audio adapter for playback;

video sending means for sending the sequence of video fields stored in the system to the video adapter for playback;

means for synchronously starting said audio sending means and said video sending means;

means for counting the video fields as each video field is processed at the video adapter for playback and providing an actual count of processed video fields;

said audio adapter providing an audio completed signal after each audio field has been processed by the adapter;

means responsive to the audio completed signal for comparing the sync count of the processed audio field with the actual count from said counting means of the processed video field; and

means for synchronizing the playback of the audio fields and video fields based on a comparison of the sync count of the processed audio field with the actual count.

6. The apparatus of claim 5 wherein said synchronizing means comprises:

next video means for getting a next video field for said video sending means if the sync count and the actual count compare equal;

repeat means for getting the most recently processed video field for said video sending means if the actual count is greater that the sync count; and

means for skipping a number of video fields and said next video means getting a next field after the skipped video fields for the video sending means if the actual count is greater that the sync count.

7. The apparatus of claim 5 wherein said comparing means comprises:

means for subtracting the actual count from the sync count and providing a skip field count;

means for testing whether the skip field count is positive, negative or zero.

8. The apparatus of claim 7 wherein said synchronizing means comprises:

next video means for getting a next video field for said video sending means if the skip field count is zero;

repeat means for getting the most recently processed video field for said video sending means if the skip field count is negative;

means for skipping a number of video fields equal to the skip field count if the skip field count is positive;

said next video means getting a next field after the skipped video fields for the video sending means.

9. In a computing system for generating audio/visual presentations, an audio/visual-synchronizer for synchronizing the playback of video data fields with audio data fields, said computing system having a processor, storage devices, an audio adapter with audio device for playing back audio fields, and a video adapter with video device for playing back video fields, and said synchronizer comprising:

the processor operating with said storage devices for synchronously recording said video fields and said audio fields and appending to each audio field a sync count equal to a video field count, said video field count is equal to the number of video fields recorded when said processor records each video field at a storage device;

the processor initiating an audio/visual presentation by operating with the storage devices and the audio and video adapters to retrieve concurrently a sequence of audio fields with appended sync counts for the audio adapter and a sequence of video fields for the video adapter;

the audio adapter processing for presentation through the audio device each audio field retrieved by the processor;

the video adapter processing for presentation through the audio device each video field retrieved by the processor;

the video adapter providing a video completed signal after each video field has been processed by the video adapter;

the processor incrementing an actual count in response to each video completed signal;

the audio adapter providing an audio completed signal after each audio field has been processed by the audio adapter;

the processor responsive to the audio completed signal for comparing the sync count of the processed audio field with the actual count of the processed video field; and

the processor synchronizing the retrieval of the audio fields and video fields for processing at the audio adapter and video adapter at each occurrence of the audio completed signal.

10. The synchronizer of claim 9 wherein the synchronous recording structure comprises:

the processor operating with storage devices for getting and saving a sequence of video fields in the computing system for subsequent playback in an audio/visual presentation;

the processor operating with storage devices for retrieving and saving audio fields in parallel with the processor operating with the storage devices to get and save video fields;

the processor for counting each video field to generate a sync count, as the video field is saved, and for appending the sync count to each audio field when the audio field is saved.

11. The synchronizer of claim 10 wherein said synchronizing structure comprises:

the processor for getting a next video field for processing by the video adapter if the sync count and actual count compare equal;

the processor for getting the most recently processed video field for the video adapter if the actual count is greater that the sync count; and

the processor for skipping a number of video fields and getting a next field after the skipped fields for the video adapter if the actual count is greater that the sync count.

12. The synchronizer of claim 9 wherein said comparing structure comprises:

the processor for subtracting the actual count from the sync count and providing a skip field count;

the processor for testing whether the skip field count is positive, negative or zero.

13. The synchronizer of claim 12 wherein said synchronizing structure comprises:

the processor for getting a next video field for the video adapter if the skip field count is zero;

the processor for getting the most recently processed video field for the video adapter if the skip field count is negative;

the processor for skipping a number of video fields equal to the skip field count if the skip field count is positive;

the processor for getting a next field after the skipped video fields for the video adapter.