Variable rate video playback with synchronized audio

• US 20040170385A1
• Filed: 02/20/2004
• Published: 09/02/2004
• Est. Priority Date: 12/05/1996
• Status: Active Grant

• Current Assignee: Vulcan Patents LLC
• Original Assignee: Interval Research Corporation
• Inventors: Ahmad, Subutai, Covell, Michele, Bhadkamkar, Neal A.
• Granted Patent: US 7,480,446 B2
• Time in Patent Office: Days
• US Class Current: 386/68
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: G11B 33/10 Indicating arrangements; Warning arrangements G11B15/04, G11B19/04, G11B27/34, G11B27/36 take precedence
• CPC Class Codes: H04N 21/426 Internal components of the client ; Characteristics thereof H04N5/44 takes precedence
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/4341 Demultiplexing of audio and video streams
• CPC Class Codes: H04N 21/439 Processing of audio elementary streams
• CPC Class Codes: H04N 21/47217 for controlling playback functions for recorded or on-demand content, e.g. using progress bars, mode or play-point indicators or bookmarks specific graphical features in visual interfaces H04N21/4312
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56
• CPC Class Codes: H04N 5/602 for digital sound signals
• CPC Class Codes: H04N 5/783 Adaptations for reproducing at a rate different from the recording rate

Abstract

The invention enables the apparent display rate of an audiovisual display to be varied. The invention can modify an original set of audio data in accordance with a target display rate, then modify a related original set of video data to conform to the modifications made to the audio data set, such that the modified audio and video data sets are synchronized. When the modified audio and video data sets so produced are used to generate an audiovisual display, the audiovisual display has an apparent display rate that approximates the target display rate. The target display rate can be faster or slower than a normal display rate at which an audiovisual display system generates an audiovisual display from the original sets of audio and video data. The target display rate can be established solely by a user instruction, by analysis of the audiovisual data, or by modification of a user-specified nominal target display rate based upon analysis of the audiovisual data. Preferably, the method for modifying the original audio data set is one that produces a modified audio data set that can be used to generate an audio display having little or no distortion.

Claims

1. A method of synchronizing a set of video data to a set of audio data that is being played at a variable rate comprising the steps of:

defining a correspondence between an original set of audio data and an original set of video data such that the original set of audio data and the original set of video data are synchronized;

creating a modified set of audio data that corresponds to the original set of audio data;

establishing a correspondence between the modified set of audio data and the original set of video data; and

creating a modified set of video data that corresponds to the original set of video data, based on the modified set of audio data and the correspondence between the modified set of audio data and the original set of video data, such that the modified set of video data is synchronized with the modified set of audio data.

2. A method of synchronizing a set of video data to a set of audio data that is being played at a variable rate comprising the steps recited in claim 1 wherein the step of defining a correspondence between the original set of audio data and the original set of video data comprises the steps of:

dividing the original set of video data into a plurality of subunits, each representing a substantially equal duration of time;

dividing the original set of audio data into a plurality of segments, each segment representing a duration of time that is approximately coincident with and substantially equal to the duration of time of a corresponding subunit of the original set of video data; and

identifying corresponding subunits of the original set of video data and segments of the original set of audio data.

3. A method of synchronizing a set of video data to a set of audio data that is being played at a variable rate comprising the steps recited in claim 1 wherein establishing a correspondence between the modified set of audio data and the original set of video data is based upon the correspondence between the modified set of audio data and the original set of audio data and the correspondence between the original set of audio data and the original set of video data.

4. A method of synchronizing a set of video data to a set of audio data that is being played at a variable rate comprising the steps recited in claim 1 wherein the step of creating a modified set of video data comprises the steps of:

identifying one or more partial or complete subunits of the original set of video data that correspond to audio segments of the modified set of audio data, based upon the correspondence between the modified set of audio data and the original set of video data; and

modifying the subunits of the original set of video data as necessary to produce the modified set of video data so that there is a one-to-one correspondence between audio segments of the modified set of audio data and subunits of the modified set of video data.

5. A method of synchronizing a set of video data to a set of audio data that is being played at a variable rate comprising the steps recited in claim 1 wherein the step of creating a modified set of video data comprises the steps of:

grouping the modified set of audio data into audio segments;

identifying one or more partial or complete subunits of the original set of video data that correspond to each of the audio segments of the modified set of audio data, based upon the correspondence between the modified set of audio data and the original set of video data; and

modifying the subunits of the original set of video data as necessary to produce the modified set of video data so that there is a one-to-one correspondence between audio segments of the modified set of audio data and subunits of the modified set of video data.

6. A method of synchronizing a set of video data to a set of audio data that is being played at a variable rate comprising the steps recited in claim 1 wherein the step of creating a modified set of video data comprises the steps of:

grouping the modified set of audio data into audio segments, each segment representing a duration of time that is approximately coincident with and substantially equal to the duration of time of a subunit of video data;

identifying one or more partial or complete subunits of the original set of video data that correspond to each of the audio segments of the modified set of audio data, based upon the correspondence between the modified set of audio data and the original set of video data; and

modifying the subunits of the original set of video data as necessary to produce the modified set of video data so that there is a one-to-one correspondence between audio segments of the modified set of audio data and subunits of the modified set of video data.

7. A method of synchronizing a set of video data to a set of audio data that is being played at a variable rate comprising the steps recited in claim 1 wherein the step of creating a modified set of video data comprises the steps of:

grouping the modified set of audio data into audio segments, each segment representing a duration of time that is approximately coincident with and substantially equal to the duration of time of a frame of video data;

identifying one or more partial or complete frames of the original set of video data that correspond to each of the audio segments of the modified set of audio data, based upon the correspondence between the modified set of audio data and the original set of video data; and

modifying the frames of the original set of video data as necessary to produce the modified set of video data so that there is a one-to-one correspondence between audio segments of the modified set of audio data and frames of the modified set of video data.

8. A method of synchronizing a set of video data to a set of audio data that is being played at a variable rate comprising the steps recited in claim 1, wherein the step of creating a modified set of video data includes the step of eliminating data from the original set of video data.

9. A method of synchronizing a set of video data to a set of audio data that is being played at a variable rate comprising the steps recited in claim 1, wherein the step of creating a modified set of video data includes the step of adding data to the original set of video data.

10. A method of synchronizing a set of video data to a set of audio data that is being played at a variable rate comprising the steps recited in claim 1, wherein the step of creating a modified set of video data includes the step of blending data from the original set of video data so that the modified set of video data has less data than the original set of video data.

11. A method of synchronizing a set of video data to a set of audio data that is being played at a variable rate comprising the steps recited in claim 1, wherein the step of creating a modified set of video data includes the step of synthesizing data, based on the data in the original set of video data, so that the modified set of video data has more data than the original set of video data.

12. A method of synchronizing a set of video data to a set of audio data that is being played at a variable rate comprising the steps recited in claim 1, further comprising the steps of:

generating an audio display from the modified set of audio data; and

generating a video display from the modified set of video data.

13. A computer program product for synchronizing a set of video data to a set of audio data that is being played at a variable rate, the computer product being embodied in a computer readable medium and comprising computer instructions for:

defining a correspondence between an original set of audio data and an original set of video data such that the original set of audio data and the original set of video data are synchronized;

creating a modified set of audio data that corresponds to the original set of audio data;

establishing a correspondence between the modified set of audio data and the original set of video data; and

creating a modified set of video data that corresponds to the original set of video data, based on the modified set of audio data and the correspondence between the modified set of audio data and the original set of video data, such that the modified set of video data is synchronized with the modified set of audio data.

14. A system for synchronizing a set of video data to a set of audio data that is being played at a variable rate comprising a processor configured to:

define a correspondence between an original set of audio data and an original set of video data such that the original set of audio data and the original set of video data are synchronized;

create a modified set of audio data that corresponds to the original set of audio data;

establish a correspondence between the modified set of audio data and the original set of video data; and

create a modified set of video data that corresponds to the original set of video data, based on the modified set of audio data and the correspondence between the modified set of audio data and the original set of video data, such that the modified set of video data is synchronized with the modified set of audio data.