Variable rate video playback with synchronized audio

• US 6,728,678 B2
• Filed: 01/07/2002
• Issued: 04/27/2004
• Est. Priority Date: 12/05/1996
• Status: Expired due to Term

• Current Assignee: Vulcan Patents LLC
• Original Assignee: Interval Research Corporation
• Inventors: Ahmad, Subutai, Covell, Michele, Bhadkamkar, Neal A.
• Primary Examiner(s): Dorvil, Richemond
• Assistant Examiner(s): Azad, Abul K.
• Application Number:
  US10/041,867
• Publication Number:
  US 20020059074A1
• Time in Patent Office: 841 Days
• Field of Search: 704/270, 704/503, 386/66, 386/54
• US Class Current: 704/270
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: G11B 33/10 Indicating arrangements; Warning arrangements G11B15/04, G11B19/04, G11B27/34, G11B27/36 take precedence
• CPC Class Codes: H04N 21/426 Internal components of the client ; Characteristics thereof H04N5/44 takes precedence
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/4341 Demultiplexing of audio and video streams
• CPC Class Codes: H04N 21/439 Processing of audio elementary streams
• CPC Class Codes: H04N 21/47217 for controlling playback functions for recorded or on-demand content, e.g. using progress bars, mode or play-point indicators or bookmarks specific graphical features in visual interfaces H04N21/4312
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56
• CPC Class Codes: H04N 5/602 for digital sound signals
• CPC Class Codes: H04N 5/783 Adaptations for reproducing at a rate different from the recording rate

Abstract

The invention enables the apparent display rate of an audiovisual display to be varied. The invention can modify an original set of audio data in accordance with a target display rate, then modify a related original set of video data to conform to the modifications made to the audio data set, such that the modified audio and video data sets are synchronized. When the modified audio and video data sets so produced are used to generate an audiovisual display, the audiovisual display has an apparent display rate that approximates the target display rate. The target display rate can be faster or slower than a normal display rate at which an audiovisual display system generates an audiovisual display from the original sets of audio and video data. The target display rate can be established solely by a user instruction, by analysis of the audiovisual data, or by modification of a user-specified nominal target display rate based upon analysis of the audiovisual data. Preferably, the method for modifying the original audio data set is one that produces a modified audio data set that can be used to generate an audio display having little or no distortion.

Claims

1. For use with an audiovisual display system in which an original set of audio data and a related original set of video data can be used to generate an audiovisual display at a normal display rate, a method for enabling the apparent display rate of the audiovisual display to be varied from the normal display rate, the method comprising the steps of:

defining a correspondence between the original set of audio data and the original set of video data such that the original set of audio data is synchronized with the original set of video data;

determining a target display rate or rates for the audiovisual display including evaluating the audio and/or video data to automatically determine the value of the target display rate including;

analyzing the original set of audio data including ascertaining the stress with which spoken portions of the audio data are uttered; and

calculating the target display rate based upon the analysis of the original set of audio data including calculating the target display rate or rates based upon the relative stresses of the spoken portions of the audio data;

creating a blended audio sample, wherein the blended audio sample is associated with a portion of a first audio sample and a portion of an adjacent second audio sample;

creating a modified set of audio data, based upon the target display rate or rates and an evaluation of the content of the original set of audio data, that corresponds to the original set of audio data; and

creating a modified set of video data, wherein the modified set of video data is associated with the blended audio sample, and wherein the modified set of video data is based upon the modified set of audio data, the correspondence between the modified set of audio data and the original set of audio data, and the correspondence between the original set of audio data and the original set of video data such that the modified set of video data is synchronized with the modified set of audio data.

2. A method as in recited in claim 1, wherein the step of ascertaining stress further comprises the step of computing energy terms for the spoken portions of the audio data.

3. For use with an audiovisual display system in which an original set of audio data and a related original set of video data can be used to generate an audiovisual display at a normal display rate, a method for enabling the apparent display rate of the audiovisual display to be varied from the normal display rate, the method comprising the steps of:

defining a correspondence between the original set of audio data and the original set of video data such that the original set of audio data is synchronized with the original set of video data;

determining a target display rate or rates for the audiovisual display, including evaluating the audio and/or video data to automatically determine the value of the target display rate, including;

analyzing the original set of audio data, including ascertaining the speaking rate at which spoken portions of the audio data are uttered; and

calculating the target display rate based upon the analysis of the original set of audio data, including calculating the target display rate or rates based upon the relative speeds of the spoken portions of the audio data;

creating a blended audio sample, wherein the blended audio sample is associated with a portion of a first audio sample and a portion of an adjacent second audio sample;

creating a modified set of audio data, based upon the target display rate or rates and an evaluation of the content of the original set of audio data, that corresponds to the original set of audio data; and

creating a modified set of video data, wherein the modified set of video data is associated with the blended audio sample, and wherein the modified set of video data is based upon the modified set of audio data, the correspondence between the modified set of audio data and the original set of audio data, and the correspondence between the original set of audio data and the original set of video data such that the modified set of video data is synchronized with the modified set of audio data.

4. A method as in recited claim 3, wherein the step of ascertaining speaking rates further comprises the step of ascertaining spectral changes in the spoken portions of the audio data.

5. For use with an audiovisual display system in which an original set of audio data and a related original set of video data can be used to generate an audiovisual display at a normal display rate, a method for enabling the apparent display rate of the audiovisual display to be varied from the normal display rate, the method comprising the steps of:

defining a correspondence between the original set of audio data and the original set of video data such that the original set of audio data is synchronized with the original set of video data;

determining a target display rate or rates for the audiovisual display, including evaluating the audio and/or video data to automatically determine the value of the target display rate, including;

analyzing the original set of audio data, including;

ascertaining the stress with which spoken portions of the audio data are uttered;

ascertaining the speaking rate at which spoken portions of the audio data are uttered; and

combining corresponding stresses and speaking rates to produce audio tension values for the spoken portions; and

calculating the target display rate based upon the analysis of the original set of audio data, including calculating the target display rate or rates based upon the audio tension values of the spoken portions of the audio data;

creating a blended audio sample, wherein the blended audio sample is associated with a portion of a first audio sample and a portion of an adjacent second audio sample;

creating a modified set of audio data, based upon the target display rate or rates and an evaluation of the content of the original set of audio data, that corresponds to the original set of audio data; and

creating a modified set of video data, wherein the modified set of video data is associated with the blended audio sample, and wherein the modified set of video data is based upon the modified set of audio data, the correspondence between the modified set of audio data and the original set of audio data, and the correspondence between the original set of audio data and the original set of video data such that the modified set of video data is synchronized with the modified set of audio data.

6. A method as recited in claim 5, further comprising the step of ascertaining the value of a nominal target display rate specified by a user of the audiovisual display system, wherein the step of calculating further comprises the step of combining the audio tension values with the nominal target display rate to produce the target display rate.

7. For use with an audiovisual display system in which an original set of audio data and a related original set of video data can be used to generate an audiovisual display at a normal display rate, a method for enabling the apparent display rate of the audiovisual display to be varied from the normal display rate, the method comprising the steps of:

defining a correspondence between the original set of audio data and the original set of video data such that the original set of audio data is synchronized with the original set of video data;

determining a target display rate or rates for the audiovisual display;

creating a blended audio sample, wherein the blended audio sample is associated with a portion of a first audio sample and a portion of an adjacent second audio sample;

creating a modified set of audio data, based upon the target display rate or rates and an evaluation of the content of the original set of audio data, that corresponds to the original set of audio data, including;

(i) dividing the original set of audio data into a plurality of segments, each segment representing a contiguous portion of the original set of audio data that occurs during a specified duration of time, each segment being adjacent to one or two other segments such that there are no gaps between segments and adjacent segments do not overlap;

(ii) selecting a first segment;

(iii) selecting a second segment, the second segment temporally adjacent to the first segment;

(iv) overlapping an end portion of the first segment with an end portion of the second segment that is adjacent to the first segment, the end portion of the first segment including first segment overlap data and the end portion of the second segment including second segment overlap data;

(v) identifying as part of the modified set of audio data the audio data from the first segment that is not part of the first segment overlap data; and

(vi) blending corresponding first segment overlap data and second segment overlap data; and

(vii) determining whether there are additional segments in the original set of audio data that have not been overlapped with an adjacent segment, wherein;

if there are additional segments, the following steps are further performed;

(viii) combining the blended overlap data with the audio data from the second segment that is not part of the second segment overlap data;

(ix) selecting the combined data as a new first segment; and

(x) selecting a new second segment that is temporally adjacent to the new first segment and that has not previously been selected as a segment; and

(xi) repeating steps (i) through (vii); and

if there are not additional segments, the following step is further performed;

(xii) identifying as part of the modified set of audio data the blended data and the audio data from the second segment that is not part of the second segment overlap data;

and creating a modified set of video data, wherein the modified set of video data is associated with the blended audio sample, and wherein the modified set of video data is based upon the modified set of audio data, the correspondence between the modified set of audio data and the original set of audio data, and the correspondence between the original set of audio data and the original set of video data such that the modified set of video data is synchronized with the modified set of audio data.

8. A computer program product for use with an audiovisual display system in which an original set of audio data and a related original set of video data can be used to generate an audiovisual display at a normal display rate, for enabling the apparent display rate of the audiovisual display to be varied from the normal display rate, the computer program product being embodied in a computer readable medium and comprising computer instructions for:

defining a correspondence between the original set of audio data and the original set of video data such that the original set of audio data is synchronized with the original set of video data;

determining a target display rate or rates for the audiovisual display including evaluating the audio and/or video data to automatically determine the value of the target display rate including;

analyzing the original set of audio data including ascertaining the stress with which spoken portions of the audio data are uttered; and

calculating the target display rate based upon the analysis of the original set of audio data including calculating the target display rate or rates based upon the relative stresses of the spoken portions of the audio data;

creating a blended audio sample, wherein the blended audio sample is associated with a portion of a first audio sample and a portion of an adjacent second audio sample;

creating a modified set of audio data, based upon the target display rate or rates and an evaluation of the content of the original set of audio data, that corresponds to the original set of audio data; and

creating a modified set of video data, wherein the modified set of video data is associated with the blended audio sample, and wherein the modified set of video data is based upon the modified set of audio data, the correspondence between the modified set of audio data and the original set of audio data, and the correspondence between the original set of audio data and the original set of video data such that the modified set of video data is synchronized with the modified set of audio data.

9. A computer program product as recited in claim 8, wherein the computer instructions for ascertaining stress further comprise computer instructions for computing energy terms for the spoken portions of the audio data.

10. A computer program product for use with an audiovisual display system in which an original set of audio data and a related original set of video data can be used to generate an audiovisual display at a normal display rate, for enabling the apparent display rate of the audiovisual display to be varied from the normal display rate, the computer program product being embodied in a computer readable medium and comprising computer instructions for:

defining a correspondence between the original set of audio data and the original set of video data such that the original set of audio data is synchronized with the original set of video data;

determining a target display rate or rates for the audiovisual display, including evaluating the audio and/or video data to automatically determine the value of the target display rate, including;

analyzing the original set of audio data, including ascertaining the speaking rate at which spoken portions of the audio data are uttered; and

calculating the target display rate based upon the analysis of the original set of audio data, including calculating the target display rate or rates based upon the relative speeds of the spoken portions of the audio data;

creating a blended audio sample, wherein the blended audio sample is associated with a portion of a first audio sample and a portion of an adjacent second audio sample;

creating a modified set of audio data, based upon the target display rate or rates and an evaluation of the content of the original set of audio data, that corresponds to the original set of audio data; and

creating a modified set of video data, wherein the modified set of video data is associated with the blended audio sample, and wherein the modified set of video data is based upon the modified set of audio data, the correspondence between the modified set of audio data and the original set of audio data, and the correspondence between the original set of audio data and the original set of video data such that the modified set of video data is synchronized with the modified set of audio data.

11. A computer program product as recited in claim 10, wherein the computer instructions for ascertaining speaking rates further comprise computer instructions for ascertaining spectral changes in the spoken portions of the audio data.

12. A computer program product for use with an audiovisual display system in which an original set of audio data and a related original set of video data can be used to generate an audiovisual display at a normal display rate, for enabling the apparent display rate of the audiovisual display to be varied from the normal display rate, the computer program product being embodied in a computer readable medium and comprising computer instructions for:

defining a correspondence between the original set of audio data and the original set of video data such that the original set of audio data is synchronized with the original set of video data;

determining a target display rate or rates for the audiovisual display, including evaluating the audio and/or video data to automatically determine the value of the target display rate, including;

analyzing the original set of audio data, including;

ascertaining the stress with which spoken portions of the audio data are uttered;

ascertaining the speaking rate at which spoken portions of the audio data are uttered; and

combining corresponding stresses and speaking rates to produce audio tension values for the spoken portions; and

calculating the target display rate based upon the analysis of the original set of audio data, including calculating the target display rate or rates based upon the audio tension values of the spoken portions of the audio data;

creating a blended audio sample, wherein the blended audio sample is associated with a portion of a first audio sample and a portion of an adjacent second audio sample;

creating a modified set of audio data, based upon the target display rate or rates and an evaluation of the content of the original set of audio data, that corresponds to the original set of audio data; and

creating a modified set of video data, wherein the modified set of video data is associated with the blended audio sample, and wherein the modified set of video data is based upon the modified set of audio data, the correspondence between the modified set of audio data and the original set of audio data, and the correspondence between the original set of audio data and the original set of video data such that the modified set of video data is synchronized with the modified set of audio data.

13. A computer program product as recited in claim 12, the computer program product further comprising computer instructions for ascertaining the value of a nominal target display rate specified by a user of the audiovisual display system, wherein the computer instructions for calculating further comprise computer instructions for combining the audio tension values with the nominal target display rate to produce the target display rate.

14. A computer program product for use with an audiovisual display system in which an original set of audio data and a related original set of video data can be used to generate an audiovisual display at a normal display rate, for enabling the apparent display rate of the audiovisual display to be varied from the normal display rate, the computer program product being embodied in a computer readable medium and comprising computer instructions for:

defining a correspondence between the original set of audio data and the original set of video data such that the original set of audio data is synchronized with the original set of video data;

determining a target display rate or rates for the audiovisual display;

creating a blended audio sample, wherein the blended audio sample is associated with a portion of a first audio sample and a portion of an adjacent second audio sample;

creating a modified set of audio data, based upon the target display rate or rates and an evaluation of the content of the original set of audio data, that corresponds to the original set of audio data, including;

(i) dividing the original set of audio data into a plurality of segments, each segment representing a contiguous portion of the original set of audio data that occurs during a specified duration of time, each segment being adjacent to one or two other segments such that there are no gaps between segments and adjacent segments do not overlap;

(ii) selecting a first segment;

(iii) selecting a second segment, the second segment temporally adjacent to the first segment;

(iv) overlapping an end portion of the first segment with an end portion of the second segment that is adjacent to the first segment, the end portion of the first segment including first segment overlap data and the end portion of the second segment including second segment overlap data;

(v) identifying as part of the modified set of audio data the audio data from the first segment that is not part of the first segment overlap data; and

(vi) blending corresponding first segment overlap data and second segment overlap data; and

(vii) determining whether there are additional segments in the original set of audio data that have not been overlapped with an adjacent segment, wherein;

if there are additional segments, the following steps are further performed;

(viii) combining the blended overlap data with the audio data from the second segment that is not part of the second segment overlap data;

(ix) selecting the combined data as a new first segment; and

(x) selecting a new second segment that is temporally adjacent to the new first segment and that has not previously been selected as a segment; and

(xi) repeating steps (i) through (vii); and

if there are not additional segments, the following step is further performed;

(xii) identifying as part of the modified set of audio data the blended data and the audio data from the second segment that is not part of the second segment overlap data;

and creating a modified set of video data, wherein the modified set of video data is associated with the blended audio sample, and wherein the modified set of video data is based upon the modified set of audio data, the correspondence between the modified set of audio data and the original set of audio data, and the correspondence between the original set of audio data and the original set of video data such that the modified set of video data is synchronized with the modified set of audio data.