Performance retiming effects on synchronized data in an editing system

• US 7,194,676 B2
• Filed: 03/01/2002
• Issued: 03/20/2007
• Est. Priority Date: 03/01/2002
• Status: Expired due to Term

• Current Assignee: Avid Technology Incorporated
• Original Assignee: Avid Technology Incorporated
• Inventors: Cornog, Katherine H., Fayan, Randy M.
• Primary Examiner(s): Hong; Stephen
• Assistant Examiner(s): Stork; Kyle R.
• Application Number:
  US10/086,478
• Publication Number:
  US 20030164845A1
• Time in Patent Office: 1,845 Days
• Field of Search: 715/500.1, 715/501.1, 715/513, 715/517, 715/723, 715/724, 715/725, 715/726, 715/523, 715/524, 715/525, 715/526, 715/530, 715/500, 348/194
• US Class Current: 715/203
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: H04N 5/262 Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects

Abstract

A retiming function that defines a rampable retiming effect is used to generate new audio and video samples at appropriate output times. In particular, for each output time, a corresponding input time is determined from the output time by using the retiming function. The retiming function may be a speed curve, a position curve that maps output times to input times directly or a mapping defining correspondence times between points in the video data and points in the audio data. An output sample is computed for the output time based on at least the data in the neighborhood of the corresponding input time, using a resampling function for the type of media data. Synchronization is achieved by ensuring that the input times determined to correspond to output times for video samples correspond to the input times determined to correspond to the same output times for audio samples.

Claims

1. In a system for editing an audiovisual work, a method for producing a retiming effect on a clip of synchronized audio data and video data, wherein the audio data comprises a sequence of audio samples wherein each audio sample has a corresponding input time, and wherein the video data comprises a sequence of video samples wherein each video sample has a corresponding input time, to produce a retimed clip of synchronized audio data and video data in the audiovisual work, wherein in the retimed clip the audio data comprises a sequence of audio samples wherein each audio sample has a corresponding output time, and wherein thc video data comprises a sequence of video samples wherein each video sample has a corresponding output time, comprising:

associating a definition of a retiming function for a rampable retirning effect that maps output times to input times with the clip of synchronized audio data and video data;

processing the synchronized audio data and video data according to the retiming function to produce the rctimed clip, wherein processing comprises;

for each output time for an audio sample,

determining a corresponding input time from the output time using the retiming function; and

computing an output audio sample for the output time by applying an audio resampling function to a plurality of innut audio samples from points in time surrounding the corresponding input time, wherein the audio resampling function combines information from the plurality of input audio samples to produce the output audio sample; and

for each output time for a video sample,

determining a corresponding input time from the output time using the retiming function, such that input times determined for output times for video samples correspond to input times determined for the same output times for audio samples; and

computing an output video sample for the output time by applying a video resampling function to a plurality of input video samples from points in time surrounding the corresponding input time, wherein the video resampling function combines information from the plurality of input video samples to produce the output video sample.

2. The method of claim 1, wherein the retiming function is defined as a speed curve, and wherein the speed curve is integrated to determine the input times from output limes for both video data and audio data, wherein a step size used to compute an integral of the speed curve is less than or equal to a step size corresponding to a sampling rate of the audio data.

3. The method of claim 1, wherein determining the output audio sample uses a time-sealing function.

4. The method of claim 3, wherein determining the output video sample uses motion based interpolation.

5. The method of claim 3, wherein determining the output video sample uses blended frames.

6. The method of claim 1, wherein determining the output audio sample uses a resampling function with pitch shifting.

7. The method of claim 6, wherein determining the output video sample uses motion based interpolation.

8. The method of claim 6, wherein determining the output video sample uses blended frames.

9. The method of claim 1, wherein an input time determined for any output time for a video sample is identical to an input time determined for the same output time for an audio sample.

10. The method of claim 1, wherein an input time determined for each output time for video samples is offset from an input time determined for the same output time for audio samples.

11. The method of claim 1, wherein the retiming function comprises a mapping of a plurality of video events in the input video data and a corresponding plurality of audio events in the input audio data to a corresponding plurality of output times in the retimed clip.

12. The method of claim 1, further comprising computing a position curve for audio from the mapping, and whcrein determining an input time from an output time for an audio sample uses the position curve.

13. The method of claim 1, further comprising computing a position curve for video from the mapping, and wherein determining an input time from an output time for a video sample uses the position curve.

14. The method of claim 1, further comprising receiving an indication of the mapping by:

presenting a graphical user interface including a video track, an audio track and an output track;

receiving an indication of a video event in the video data on the video track through an input device;

receiving an indication of an audio event in the audio data on the audio track through an input device;

receiving an indication of an output time in the retimed clip on the output track through the input device; and

maintaining in formation indicating a correspondence between the indicated video event, the indicated audio event and the indicated output time.

15. The method of claim 14, further comprising computing a position curve for audio from the mapping, and wherein determining an input time from an output time for an audio sample uses the position curve.

16. The method of claim 15, further comprising computing a position curve for video from the mapping, and wherein determining an input time from an output time for a video sample uses the position curve.

17. The method of claim 14, further comprising computing a position curve for video from the mapping, and wherein determining an input time from an output time for a video sample uses the position curve.

18. A computer program product, comprising:

a computer readable medium;

computer program instructions stored in the computer readable medium that, when executed by a computer, instruct the computer to perform a method for producing a retiming effect on a clip of synchronized audio data and video data, wherein the audio data comprises a sequence of audio samples wherein each audio sample has a corresponding input time, and wherein the video data comprises a sequence of video samples wherein each video sample has a corresponding input time, to produce a retimed clip of synchronized audio data and video data in the audiovisual work, wherein in the retimed clip the audio data comprises a sequence of audio samples wherein each audio sample has a corresponding output time, and wherein the video data comprises a sequence of video samples wherein each video sample has a corresponding output time, comprising;

associating a definition of a retiming function for a rampable retiming effect that maps output times to input times with the clip of synchronized audio data and video data;

processing the synchronized audio data and video data according to the retiming function to produce the retiming clip, wherein processing comprises;

for each output time for an audio sample,

determining a corresponding input time from the output time using the retiming function; and

computing an output audio sample for the output time by applying an audio resampling function to a plurality of input audio samples from points in time surrouding the corresponding input time, wherein the audio resampling function combines information from the plurality of input audio samples to produce the output audio sample; and

for each output time for a video sample,

determining a corresponding input time from the output time using the retiming function, such that input times determined for output times for video samples correspond to input times determined for the same output times for audio samples; and

computing an output video sample for the output time by applying a video resampling function to a plurality of input video samples from points in time surrounding the corresponding input time, wherein the video resampling function combines information from, the plurality of input video samples to produce the output video sample.

19. An editing system for editing an audiovisual work and for producing a retiming effect on a clip of synchronized audio data and video data, wherein the audio data comprises a sequence of audio samples wherein each audio sample has a corresponding input time, and wherein the video data comprises a sequence of video samples wherein each video sample has a corresponding input time, to produce a retimed clip of synchronized audio data and video data in the audiovisual work, wherein in the retimed clip the audio data comprises a sequence of audio samples wherein each audio sample has a corresponding output time, and wherein the video data comprises a sequence of video samples wherein each video sample has a corresponding output time, comprising:

means for relating a definition of a retiming function for a rampable retiming effect that maps output times to input times with the clip of synchronized audio data and video data;

means for processing the synchronized audio data and video data according to the retiming function to produce the retimed clip, comprising;

means for computing, for each output time for an audio sample, a corresponding input time from the output time using the retiming function;

means for computing an output audio sample for the output time by applying an audio resampling function to a plurality of input audio samples from points in time surrounding the corresponding input time, wherein the audio resampling function combines information from the plurality of input audio samples to produce the output audio sample;

means for computing, for each output time for a video sample, a corresponding input time from the output time using the retiming function, such that input times determined for output times for video samples correspond to input times determined for the same output times for audio samples; and

means for computing an output video sample for the output time by applying a video resampling function to a plurality of input video samples from points in time surrounding the corresponding input time, wherein the video resampling function combines information from the plurality of input video samples to produce the output video sample.

20. An editing system for editing an audiovisual work and for producing a retiming effect on a clip of synchronized audio dala and video data, wherein the audio data comprises a sequence audio samples wherein each audio sample has a corresponding input time, and wherein the video data comprises a sequence of video samples wherein each video sample has a corresponding input time, to produce a retimed clip of synchronized audio data and video data in the audiovisual work, wherein in the retimed clip the audio data comprises a sequence of audio samples wherein each audio sample has a corresponding output time, and wherein the video data comprises a sequence of video samples wherein each video sample has a corresponding output time, comprising:

an editing interface allowing a user to associate a definition of a retiming function for a rampable retiming effect that maps output times to input times with the clip of synchronized audio data and video data;

an audio retiming module having an input for receiving the definition of the retiming function and an input for receiving the audio data, and an output providing retimed audio data such that, for each output time for an audio sample, an output audio sample is computed by applying an audio resampling function to a plurality of input audio samples from points in time surrounding an input time mapped to the output time by the retiming function, wherein the audio resampling function combines information from the plurality of input audio samples to produce the output audio sample; and

a video retiming module having an input for receiving the definition of the retiming function and an input for receiving the video data, and an output providing retimed video data such that, for each output time for a video sample, an output video sample is computed by applying a video resampling function to a plurality of input video samples from points in time surrounding an input time mapped to the output time by the retiming function, wherein the video resampling function combines information from the p1urality of input video samples to produce the output video sample, wherein input times determined for output times for video samples correspond to input times determined for the same output times for audio samples.

21. A digital entertainment product, comprising:

a computer readable medium;

information stored on the computer readable medium that, when read by a computer, indicates to the computer a retimed clip of synchronized audio data and video data, produced according to a process for retiming a clip of synchronized audio data and video data, wherein the audio data comprises a sequence of audio samples wherein each audio sample has a corresponding input time, and wherein the video data comprises a sequence of video samples wherein each video sample has a corresponding input time, to produce the retimed clip of synchronized audio data and video data in the audiovisual work, wherein in the retimed clip the audio data comprises a sequence of audio samples wherein each audio sample has a corresponding output time, and wherein the video data comprises a sequence of video samples wherein each video sample has a corresponding output time, the process comprising;

associating a definition of a retiming function for a rampable retiming effect that maps output times to input times with the clip of synchronized audio data and video data;

processing the synchronized audio data and video data according to the retiming function to produce the retimed clip, wherein processing comprises;

for each output time for an audio sample,

determining a corresponding input time from the output time using the retiming function; and

computing an output audio sample for the output time by applying an audio resampling function to a plurality of input audio samples from points in time surrounding the corresponding input time, wherein the audio resampling function combines information from the plurality of input audio samples to produce the output audio sample; and

for each output time for a video sample,

determining a corresponding input time from the output time using the retiming function, such that input times determined for output times for video samples correspond to input times determined for the same output times for audio samples; and

computing an output video sample for the output time by applying a video resampling function to plurality of input video samples from points in time surrounding the corresponding input time, wherein the video resampling function combines information from the plurality of input video samples to produce the output video sample.

22. In a system for editing an audiovisual work including a sequence of a plurality of clips of audiovisual data, a method for producing a retiming effect on a clip of synchronized audio data and video data, wherein the audio data comprises a sequence of audio samples wherein each audio sample has a corresponding input time, and wherein the video data comprises a sequence of video samples wherein each video sample has a corresponding input time, to produce a retimed clip of synchronized audio data and video data in the audiovisual work, wherein in the retimed clip thc audio data comprises a sequence of audio samples wherein each audio sample has a corresponding output time, and wherein the video data comprises a sequence of video samples wherein each video sample has a corresponding output time, comprising:

associating a definition of a retiming function for a rampable retiming effect that maps output times to input times with the clip of synchronized audio data and video data;

processing the video data of the clip according to the retiming function to produce a retimed video clip, wherein processing comprises;

for each output time for a video sample,

determining a corresponding input time from the output time using the retiming function; and

computing an output video sample for the output time by applying a video resampling function to a plurality of input video samples from points in time surrounding the corresponding input time, wherein the video resampling function combines information from the plurality of input video samples to produce the output video sample;

placing the retimed video clip in the audiovisual work; and

exporting the audiovisual work to an audio editing system, including the definition of the retiming function, for processing the audio data of the clip according to the retiming function, such that an input time determined for each output time for video samples corresponds to an input time determined for the same output time for audio samples.

23. In a system for editing an audiovisual work including a sequence of a plurality of clips of audiovisual data, a method for producing a retiming effect on a clip of synchronized audio data and video data, wherein the audio data comprises a sequence of audio samples wherein each audio sample has a corresponding input time, and wherein the video data comprises a sequence of video samples wherein each video sample has a corresponding input time, to produce a retimed clip of synchronized audio and video data in the audiovisual work, wherein in the retimed clip the audio data comprises a senuence of audio samples wherein each audio sample has a corresponding output time, and wherein the video data comprises a sequence of video samiles wherein each video sample has a corresponding output time, comprising:

receiving a definition of a retiming function for a rampable retiming effect that maps output times to input times associated with the clip of synchronized audio data and video data;

receiving the audio data;

receiving the audiovisual work including a retimed video clip processed according to the retiming function;

processing the audio data according to the retiming function to produce a retimed audio clip, wherein processing comprises;

for each output time for an audio sample,

determining a corresponding input time from the output time using the retiming function, such that an input time determined for each output time for video samples corresponds to an input time determined for the same output time for audio samples; and

computing an output audio sample for the output time by applying an audio resampling function to a plurality of input audio samples from points in time surrounding the corresponding input time, wherein the audio resampling function combines information from the plurality of input audio samples to produce the output audio sample; and

synchronizing the retimed audio clip with the retimed video clip in the audiovisual work.

24. In a system for editing an audiovisual work including a sequence of a plurality of clips of audiovisual data, a method for producing a retiming effect on a clip of synchronized audio data and video data, wherein the audio data comprises a sequence of audio samples wherein cach audio sample has a corresponding input time, and wherein the video data comprises a sequence of video samples wherein each video sample has a corresponding input time, to produce a retimed clip of synchronized audio and video data in the audiovisual work, wherein in the retimed clip the audio data comprises a sequence of audio samples wherein each audio sample has a corresponding output time, and wherein the video data comprises a sequence of video samples wherein each video sample has a corresponding output time, comprising:

receiving a definition of a retiming function for a rampable retiming effect that maps output times to input times associated with the clip of synchronized audio data and video data;

receiving the video data;

receiving the audiovisual work including a retimed audio clip processed according to the retiming function;

processing the video data according to the retiming function to produce a retimed video clip, wherein processing comprises;

for each output time for a video sample,

determining a corresponding input time from the output time for the retiming function, such that an input time determined for each output time for video samples corresponds to an input time determined for the same output time for audio samples; and

computing an output video sample for the output time by applying a video resampling function to a plurality of input video samples from points in time surrounding the corresponding input time, wherein the video resampling function combines information from the plurality of input video samples to produce the output video sample; and

synchronizing the retimed video clip with the retimed audio clip in the audiovisual work.

25. In a system for editing an audiovisual work, a method for defining a retiming effect applied to audio data and video data, wherein the audio data comprises a sequence of audio samples wherein each audio sample has a corresponding input time, and wherein the video data comprises a sequence of video samples wherein each video sample has a corresponding input time, to produce a retimed clip of synchronized audio and video data in the audiovisual work, wherein in the retimed clip the audio data comprises a scouence of audio samples wherein each audio sample has a corresponding output time, and wherein the video data comprises a sequence of video samples wherein each video sample has a corresponding output time, comprising:

associating a mapping of a plurality of video events in the video data and a corresponding plurality of audio events in the audio data to a corresponding plurality of output times in the retimed clip;

processing the audio data according to an audio resampling function that generates each output audio sample, for each output time for the retimed clip, from a plurality of input audio samples from points in time in the audio data surrounding an input time mapped to the output time, to produce retimed audio data, wherein the audio resampling function combines information from the plurality of input audio samples to produce the output audio sample;

processing the video data according to a video resampling function that generates each output video sample, for each output time for the retimed clip, from a plurality of input video samples from points in time in the video data surrounding an input time mapped to the output time, to produce retimed video data, wherein the video resampling function combines information from the plurality of input video samples to produce the output video sample; and

placing the retimed audio data and retimed video data in the audiovisual work as the retimed clip of synchronized audio and video data.

26. The method of claim 25, wherein processing the audio data comprises

computing a position curve for audio from the mapping;

for each output time for an audio sample,

determining the corresponding input time from the output time using the position curve; and

computing an output audio sample for the output time based on the plurality of input audio samples from points in time surrounding the corresponding input time using the audio resampling function.

27. The method of claim 26, wherein processing the video data comprises

computing a position curve for video from the mapping;

for each output time for a video sample,

determining the corresponding input time from the output time using the position curve; and

computing an output video sample for the output time based on the plurality of input video samples from points in time surrounding the corresponding input time using the video resampling function.

28. The method of claim 25, wherein processing the video data comprises

computing a position curve for video from the mapping;

for each output time for a video sample,

determining the corresponding input time from the output time using the position curve; and

computing an output video sample for the output time based on the plurality of input video samples from points in time surrounding the corresponding input time using the video resampling function.

29. The method of claim 25, further comprising receiving an indication of the mapping by:

presenting a graphical user interface including a video track, an audio track and an output track;

receiving an indication of a video evcnt in the video data on the video track through an input device;

receiving an indication of an audio event in the audio data on the audio track through an input device;

receiving an indication of an output time in the retimed clip on the output track through the input device; and

maintaining information indicating a correspondence between the indicated video event, the indicated audio event and thc indicated output time.

30. A computer program product, comprising:

a computer readable medium;

computer program instructions stored on the computer readable medium that, when executed by a computer, instructs the computer to perform a method for defining a retiming effect applied to audio data and video data, wherein the audio data comprises a sequence of audio samples wherein each audio sample has a corresponding input time, and wherein the video data comprises a sequence of video samples wherein each video sample has a corresponding input time, to produce a retimed clip of synchronized audio and video data in the audiovisual work, wherein in the retimed clip the audio data comprises a sequence of audio samples wherein each audio sample has a corresponding output time, and wherein the video data comprises a sequence of video samples wherein each video sample has a corresponding output time, comprising;

associating a mapping of a plurality of video events in the video data and a corresponding plurality of audio events in the audio data to a corresponding plurality of output times in the retimed clip;

processing the audio data according to an audio resampling function that generates each output audio sample, for each output time for the retimed clip, from a plurality of input audio samples from points in time in the audio data surrounding an input time mapped to tho output time, to produce retimed audio data, wherein the audio resampling function combines information from the plurality of input audio samples to produce the output audio sample;

processing the video data according to a video resampling function that generates each output video sample, for each output time for the retimed clip, from a plurality of input video samples from points in time in the video data surrounding an input time mapped to the output time, to produce retimed video data, wherein the video resampling function combines information from the plurality of input video samples to produce the output video sample; and

placing the retimed audio data and retimed video data in thc audiovisual work as the retimed clip of synchronized audio and video data.

31. An editing system for defining a retiming effect applied to audio data and video data, wherein the audio data comprises a sequence of audio samples wherein each audio sample has a corresponding input time, and wherein the video data comprises a sequence of video samples wherein each video sample has a corresponding input time, to produce a retimed clip of synchronized audio and video data in the audiovisual work, wherein in the retimed clip the audio data comprises a sequence of audio samples wherein each audio sample has a corresponding output time, and wherein the video data comprises a sequence of video samples wherein each video sample has a corresponding output time, comprising:

means for mapping a plurality of video events in the video data and a corresponding plurality of audio events in the audio data to a corresponding plurality of output times in the retimed clip;

means for processing the audio data according to an audio resampling function that generates each output audio sample, for each output time for the retimed clip, from a plurality of input audio samples fiom points in time in the audio data surrounding an input time mapped to the output time, to produce retimed audio data, wherein the audio resampling function combines information from the plurality of input audio samples to produce the output audio sample;

means for processing the video data according to a video resampling function that generates each output video sample, for each output time for the retimed clip, from a plurality of input video samples from points in time in the video data surrounding an input time mapped to the output time, to produce retimed video data, wherein the video resampling function combines information from the plurality of input video samples to produce the output video sample; and

means for inserting the retimed audio data and retimed video data in the audiovisual work as the retimed clip of synchronized audio and video data.

32. A method for applying a rampable retiming effect to synchronized streams of temporal media data, include a first stream of samples of a first media type wherein each sample has a corresponding input time, and a second stream of samples of a second media type different from the first media type, wherein each sample has a corresponding input time, to produce a retimed clip of synchronized temporal media data, wherein in the retimed clip each sample of the first media type has a corresponding output time and each sample of the second media type has a corresponding output time, comprising:

associating a retiming function for the rampable retiming effect with the synchronized streams, wherein the retiming function maps output times to input times;

processing the synchronized streams according to the retiming function, wherein processing comprises;

for each output time for the first stream,

determining a corresponding input time from the output time using the retiming function; and

computing an output sample for the first stream for the output time by applying a resampling function for the first media type to a plurality of input samples of the first media type from points in time surrounding the corresponding input time, wherein the resampling function for the first media type combines information from the plurality of input samples of the first media type to produce the output sample of the first media type; and

for each output time for the second stream,

determining a corresponding input time from the output time using the retiming function, such that an input time determined for an output time for a sample of the second stream corresponds to an input time determined for the same output times for a sample of the first stream; and

computing an output sample for the second stream for the output time by applying a resampling function for the second media type to a plurality of input samples of the second media type from points in time surrounding the corresponding input time, wherein the resampling function for the second media type combines information from the plurality of input samples of the second media type to produce the output sample of the second media type.

33. A method for applying a rampable retiming effect to synchronized streams of temporal media data, wherein the streams include a first stream of samples of a first media type wherein each sample has a corresponding input time, and a second stream of samples of a second media type, wherein each sample has a corresponding input time, to produce a retimed clip of synchronized temporal media data, wherein in the retimed clip each sample of the first media type has a corresponding output time and each sample of the second media type has a corresponding output time, comprising:

associating a retiming function for the rampable retiming effect with the synchronized streams, wherein the retiming function defines a mapping of output times to input times, wherein the mapping has a corresponding temporal resolution of greater than or equal to a temporal resolution of both of the synchronized streams;

processilig the synchronized streams according to the retiming function, wherein processing comprises;

for each output time for the first stream,

determining a corresponding input time from the output time using the mapping defined by the retiming function; and

computing an output sample for the first stream for the output time by applying a resampling function for the first media type to a plurality of input samples of the first media type from points in time surrounding the corresponding input time, wherein the resampling function for the first media type combines information from the plurality of input samples of the first media type to produce the output sample of the first media type; and

for each output time for the second stream,

determining a corresponding input time from the output time using the mapping defined by the retiming function; and

computing an output sample for the second stream for the output time by applying a resampling function for the second media type to a plurality of input samples of the second media type from points in time surrounding the corresponding input time, wherein the resampling function for the second media type combines information from the plurality of input samples of the second media type to produce the output sample of the second media type.

34. A method for applying a rampable retiming effect to temporal media data synchronized with metadata, wherein the temporal media data includes a sequence of samples wherein each sample has a corresponding input time, to produce a retimed clip, wherein the retimed clip includes a sequence of samples of temporal media data wherein each sample has a corresponding output time, comprising:

associating a retiming function for the rampable retiming effect with the temporal media data, wherein the retiming function maps output times to input times;

processing the temporal media data according to the retiming function, wherein processing comprises, for each output time,

determining a corresponding input time from the output time using the retiming function; and

computing an output sample of the temporal media data for the output time by applying a resampling function for the temporal media data to a plurality of input samples of the temporal media data from points in time surrounding the corresponding input time, wherein the resampling function for the temporal media data combines information from the plurality of input samples of the temporal media data to produce the output sample of the temporal media data; and

processing the ractadata to synchronize the metadata with the retimed temporal media data.