Systems and methods for multimedia time stretching

• US 7,676,142 B1
• Filed: 06/09/2003
• Issued: 03/09/2010
• Est. Priority Date: 06/07/2002
• Status: Active Grant

• Current Assignee: Corel Corporation (Corel Holdings LP)
• Original Assignee: Corel Incorporated (Corel Holdings LP)
• Inventors: Hung, Andy Chao
• Primary Examiner(s): NGUYEN BA, HOANG VU A
• Application Number:
  US10/458,066
• Time in Patent Office: 2,465 Days
• Field of Search: 725/88, 725/102, 386/131, 386/68, 386/75, 348/446
• US Class Current: 386/233
• CPC Class Codes: G11B 27/005 Reproducing at a different information rate from the information rate of recording for television signals H04N5/783
• CPC Class Codes: H04N 21/4325 by playing back content from the storage medium reproduction of recorded television signals H04N5/76; reproduction of recorded television signals H04N9/79
• CPC Class Codes: H04N 21/4398 involving reformatting operations of audio signals details of audio signal transcoding G10L19/173
• CPC Class Codes: H04N 21/4402 involving reformatting operations of video signals for household redistribution, storage or real-time display details of conversion of video standards at pixel level H04N7/01; video transcoding H04N19/40; adapting incoming signals to the display format of the display terminal G09G5/005; media handling at the source in data packet switching networks H04L65/764
• CPC Class Codes: H04N 21/458 Scheduling content for creating a personalised stream, e.g. by combining a locally stored advertisement with an incoming stream; Updating operations, e.g. for OS modules ; time-related management operations arrangements for replacing or switching information during the broadcast or during the distribution H04H20/10
• CPC Class Codes: H04N 5/781 on disks or drums
• CPC Class Codes: H04N 5/783 Adaptations for reproducing at a rate different from the recording rate
• CPC Class Codes: H04N 5/85 on discs or drums
• CPC Class Codes: H04N 7/0127 by changing the field or frame frequency of the incoming video signal, e.g. frame rate converter
• CPC Class Codes: H04N 9/8042 involving data reduction
• CPC Class Codes: H04N 9/8063 using time division multiplex of the PCM audio and PCM video signals
• CPC Class Codes: H04N 9/8227 the additional signal being at least another television signal
• CPC Class Codes: H04N 9/8233 the additional signal being a character code signal

Abstract

The invention is related to methods and apparatus that can advantageously alter a playback rate of a multimedia presentation, such as a video clip. One embodiment of the invention permits a multimedia presentation to be sped up or slowed down with a controlled change in pitch of the sped up or slowed down audio. In one embodiment, this controlled change in the pitch permits the sped up or slowed down audio to retain a same sounding pitch as at normal playback speeds. In one embodiment, a duration is specified and playback of the video clip is advantageously sped to complete playback within the specified duration. In another embodiment, a finish by a time is specified, and the playback of the video clip is advantageously sped to complete playback by the specified time.

Claims

1. A method of altering a speed of a multimedia presentation, the method comprising:

receiving encoded source data for the multimedia presentation that has been compressed by a video compression process, where the multimedia presentation is intended to be presented at a first frame rate;

decoding visual data from the encoded source data;

wherein if the encoded source data is in a progressive format, the step of decoding includes converting said visual data in said progressive format into an interlaced format, for presentation on an interlaced format display device, and wherein if the encoded source data is in an interlaced format, the step of decoding includes converting said visual data in said interlaced format into a progressive format, for presentation on a progressive format display device;

decoding audio data from the encoded source data;

performing a time-stretching process on the decoded audio data, where the time stretching process changes a number of samples of the audio data such that the audio data can be presented at a second effective frame rate that is effectively different from the first frame rate, where a pitch of the audio data is controlled; and

providing decoded visual data and decoded audio data for presentation at the second effective frame rate, where the decoded visual data and the decoded audio data are synchronized.

2. The method as defined in claim 1, where the time-stretching process maintains a sampling rate of the audio data such that an original pitch of the audio data sounds unaltered.

3. The method as defined in claim 1, wherein the second effective frame rate is higher than the first frame rate, further comprising removing pictures from the multimedia presentation to effectively increase the second effective frame rate.

4. The method as defined in claim 1, wherein the second effective frame rate is lower than the first frame rate, further comprising repeating pictures from the multimedia presentation to effectively lower the second effective frame rate.

5. The method as defined in claim 1, wherein the video compression process comprises MPEG-2.

6. The method as defined in claim 1, wherein the method is performed in real time such that the time-stretching process is performed while the multimedia process is presented.

7. The method as defined in claim 1, wherein the time stretching process corresponds to extending time, such that a time-stretch factor α

, where the time-stretch factor α

represents a ratio between a time-stretched number of audio data samples and an original number of audio data samples in a frame of data, is greater than 1.

8. The method as defined in claim 1, wherein the time stretching process corresponds to shortening time, such that a time-stretch factor α

, where the time-stretch factor α

represents a ratio between a time-stretched number of audio data samples and an original number of audio data samples in a frame of data, is less than 1.

9. The method as defined in claim 1, wherein the encoded source data for the multimedia presentation includes time stamps indicating when decoded data is to be presented, where the first frame rate is determined by a processing of the time stamps at a first system clock speed that is specified by the video compression process, and where the second effective frame rate is determined by a processing of the time stamps at a second system clock speed that is different from the first system clock speed.

10. The method as defined in claim 1, further comprising displaying the decoded visual data on a display device.

11. The method as defined in claim 1, further comprising demultiplexing the encoded source data for the multimedia presentation from a multiplexed data channel which carries a plurality of multimedia presentations.

12. The method as defined in claim 1, further comprising reading the encoded source data from a digital versatile disc (DVD).

13. The method as defined in claim 1, further comprising disabling the time-stretching process when a time-stretch factor α

, where the time-stretch factor α

represents a ratio between a time-stretched number of audio data samples and an original number of audio data samples in a frame of data, is outside the range of about 0.5×

-2.9×

, where X relates to a playback rate, where 1×

relates to a playback rate at a speed intended by a video encoder.

14. The method as defined in claim 1, further comprising:

decoding sub-picture data from the encoded source data; and

providing decoded sub-picture data for presentation at the second effective frame rate, such that the sub-picture data is synchronized with the visual data and the audio data.

15. The method as defined in claim 1, further comprising:

decoding closed-caption data from the encoded source data; and

providing decoded closed-caption data for presentation at the second effective frame rate such that the closed-caption data is synchronized with the visual data and the audio data.

16. The method as defined in claim 1, further comprising:

receiving an indication of a selected amount of time; and

selecting the second effective frame rate such that playback can be completed within the selected time.

17. The method as defined in claim 16, wherein the selected amount of time corresponds to a “

play in”

time.

18. The method as defined in claim 16, further comprising:

providing a user interface; and

retrieving an indication from the user interface relating to the selected amount of time.

19. The method as defined in claim 16, wherein determining an amount of remaining time corresponds to using a run time for the multimedia presentation when the determination is performed before playback has initiated.

20. The method as defined in claim 16, further comprising:

determining an amount of remaining time in the multimedia presentation, where the amount of remaining time is related to playback at the first frame rate;

comparing the selected amount of time to the amount of remaining time; and

determining the second effective frame rate at least partially from the first frame rate and a ratio of the remaining time to the selected amount of time.

21. The method as defined in claim 1, further comprising:

receiving an indication of a selected time; and

selecting the second effective frame rate such that playback can be completed by the selected time.

22. The method as defined in claim 21, wherein the selected time corresponds to a desired finish time.

23. The method as defined in claim 21, further comprising:

providing a user interface, and

retrieving an indication from the user interface relating to the selected time.

24. The method as defined in claim 21, further comprising receiving an indication of an hour and a minute for the selected time.

25. The method as defined in claim 21, further comprising:

determining an amount of a first remaining time period in the multimedia presentation, where the amount of the first remaining time period is related to playback at the first frame rate;

determining an amount of a second remaining time period in the multimedia presentation, where the amount of the second remaining time period is related to playback at the second frame rate;

comparing the amount of the first remaining time period to the amount of the second remaining time period; and

determining the second frame rate at least partially from the first frame rate and a ratio of the first remaining time period to the second remaining time period.

26. The method as defined in claim 1, wherein the first frame rate corresponds to a video frame rate and the second effective frame rate corresponds to a film frame rate.

27. The method as defined in claim 1, wherein the first frame rate is about 25 frames per second (fps) and the second effective frame rate is about 24 frames per second.

28. The method as defined in claim 1, further comprising:

determining that the first frame rate is about 25 frames per second (fps);

determining that an original source for the multimedia presentation is film; and

using about 24 fps as the second effective frame rate.

29. The method as defined in claim 28, wherein a video format is PAL.

30. The method as defined in claim 28, wherein a video format is SECAM.

31. The method as defined in claim 1, further comprising:

receiving the multimedia presentation in a data stream;

monitoring an occupancy level of a decoder buffer that is adapted to temporarily store data from the data stream; and

adaptively changing the second effective frame rate in response to the occupancy level.

32. The method as defined in claim 31, wherein adaptively changing further comprises comparing an updated value for the second effective frame rate to an existing second effective frame rate, and not changing the second effective frame rate unless a difference between the updated value and the existing second effective frame rate exceeds a predetermined amount.

33. The method as defined in claim 31, further comprising:

initiating playback of the multimedia presentation earlier than indicated by the multimedia presentation, where a presentation of the multimedia presentation begins with the second effective frame rate slower than the first frame rate;

accumulating additional data in the decoder buffer as the multimedia presentation is presented at the second effective frame rate; and

presenting the multimedia presentation at the first frame rate a the buffer occupancy level has increased to a predetermined level.

34. The method as defined in claim 1, wherein the method is performed in a device powered by a battery, the method further comprising:

determining an amount of battery charge remaining in the battery; and

selecting the second effective frame rate at least partially in response to a combination of the amount of battery charge remaining and an amount of play time remaining in the multimedia presentation such that playback of the multimedia presentation can be completed before the battery runs out of charge.

35. The method as defined in claim 1, wherein the method is performed in a device coupled to an electronic network;

the method further comprising;

determining a speed of the electronic network; and

selecting the second effective frame rate at least partially in response to the speed of the electronic network.

37. The method as defined in claim 1, further comprising:

determining that an audio sound level of a portion of the media presentation is low;

determining that a motion activity of the portion of the media presentation is low; and

selectively performing the time-stretching process to speed up playback of the portion with the low audio sound level and the low motion activity.

38. The method as defined in claim 1, further comprising:

storing a time-stretch profile that varies over time; and

using the time-stretch profile to control the second effective frame rate.

39. The method as defined in claim 1, further comprising:

providing a user interface; and

retrieving an indication from the user interface relating to an amount by which the speed of the multimedia presentation is to be changed.

40. The method as defined in claim 39, further comprising providing a movable slider in the user interface, where movement of the slider changes the amount by which the speed of the multimedia presentation is to be changed.

41. The method as defined in claim 40, further comprising displaying the amount by which the speed of the multimedia presentation is to be changed in the user interface.

42. The method as defined in claim 40, further comprising computing and displaying a remaining time in the user interface in response to the movement of the slider.

43. The method as defined in claim 40, further comprising computing and displaying a finish time in the user interface in response to the movement of the slider.

44. The method as defined in claim 1, further comprising selectively performing the time-stretching process on the decoded audio, wherein selectively performing includes using different frame rates on different frames of audio data such that audio is realigned with visual images.

45. The method as defined in claim 1, further comprising:

receiving a navigation command for a requested task;

performing the requested task; and

resuming playback at the second effective frame rate.

46. The method as defined in claim 1, further comprising:

receiving a navigation command selectable from a plurality of navigation commands;

performing a task corresponding to the selected navigation command;

resuming playback at the second effective frame rate when the navigation command corresponds to a first group; and

resuming playback at the first frame rate when the navigation command corresponds to a second group.

47. The method as defined in claim 1, further comprising:

receiving a navigation command corresponding to skip;

performing the skip; and

resuming playback at the second effective frame rate.

48. The method as defined in claim 1, further comprising:

receiving a navigation command corresponding to fast forward;

performing the fast forward; and

resuming playback at the second effective frame rate.

49. The method as defined in claim 1, further comprising:

receiving a navigation command corresponding to rewind;

performing the rewind; and

resuming playback at the second effective frame rate.

50. The method as defined in claim 1, further comprising:

receiving a navigation command corresponding to pause;

performing the pause; and

resuming playback at the second effective frame rate.

51. The method as defined in claim 1, further comprising:

receiving a navigation command corresponding to slow motion;

performing the slow motion; and

resuming playback at the second effective frame rate.

52. The method as defined in claim 51, wherein the slow motion is presented at a third effective rate, further comprising performing the time-stretching process on the decoded audio data at the third effective rate, where the pitch of the audio data during the presentation of slow motion is controlled.

53. The method as defined in claim 52, where the time-stretching process maintains a sampling rate of the audio data such that an original pitch of the audio data sounds unaltered during the presentation of slow motion.

54. The method as defined in claim 1, further comprising:

receiving a navigation command corresponding to stop;

performing the stop; and

resuming playback at the first frame rate.

55. The method as defined in claim 1, further comprising:

receiving a navigation command corresponding to a change of a media selection;

performing the change of the media selection; and

playing back a new media presentation at the first frame rate.

56. The method as defined in claim 1, further comprising:

receiving a navigation command selected from the group consisting of skip, fast forward, rewind, stop, change media selection, pause, and slow motion;

performing the requested navigation command;

resuming playback at the second effective frame rate when the selected navigation command was selected from the group consisting of skip, fast forward, rewind, pause, and slow motion; and

resuming playback at the first frame rate when the selected navigation command was selected from the group consisting of stop and change media selection.

57. The method as defined in claim 1, further comprising:

receiving a navigation command for a seek in a forward direction at x speed, where x is greater than 1; and

presenting the multimedia presentation at a fast forward rate in response to the navigation command, where the fast forward rate has an effective frame rate of a product of x and the second effective frame rate.

58. The method as defined in claim 1, further comprising:

receiving a navigation command for seek in a reverse direction at x speed, where x is greater than 1; and

presenting the multimedia presentation backwards at a reverse rate in response to the navigation command, where the reverse rate has an effective frame rate of a product of x and the second effective frame rate.

59. The method as defined in claim 1, further comprising determining locations for adding pitch periods using correlation techniques.

60. The method as defined in claim 1, further comprising determining locations for removing pitch periods using correlation techniques.

61. The method as defined in claim 1, further comprising:

using computations of correlation to identify pitch periods;

searching for pitch periods using subsampling techniques; and

subsampling within the correlation calculation.

62. The method as defined in claim 1, further comprising:

using computations of correlation to identify pitch periods;

searching a coarse position grid with subsampling techniques to obtain a coarse search position; and

refining the coarse search position by searching around the coarse search position with a finer resolution than a resolution used to find the coarse search position.

63. The method as defined in claim 1, further comprising using overlapping windowing when combining pitch segments to reduce glitches and pops.

64. The method as defined in claim 1, further comprising varying a size of a search range for a pitch period, wherein varying depends on time stretch factor and pitch period size.

65. The method as defined in claim 1, further comprising varying a size for a signal segment, wherein varying depends on a time stretch factor and pitch period size.

36. The method as defined in Clam 35, wherein the electronic network includes access via the Internet.

66. A computer program product containing a computer readable storage medium with computer executable instructions embodied therein for altering a speed of an original multimedia presentation, comprising:

instructions for receiving encoded source data for the original multimedia presentation that has been compressed by a video compression process, where the multimedia presentation is intended to be presented at a first frame rate;

instructions for decoding visual data from the encoded source data;

wherein the instructions for decoding the visual data include instructions for converting said visual data if it is in a progressive format into an interlaced format, for presentation on an interlaced format display device, and instructions for converting said visual data if it is in an interlaced format into a progressive format, for presentation on a progressive format display device;

instructions for decoding audio data from the encoded source data;

instructions for performing a time-stretching process on the decoded audio data, where the time stretching process changes a number of samples of the audio data, where the time stretching process changes a number of samples of the audio data such that the audio data can be presented at a second effective frame rate that is effectively different from the first frame rate, where a pitch of the audio data is controlled; and

instructions for providing decoded visual data and decoded audio data for presentation at the second effective frame rate, where the decoded visual data and the decoded audio data are synchronized.

67. A circuit for altering a speed of a multimedia presentation comprising:

means for receiving encoded source data for the multimedia presentation that has been compressed by a video compression process, where the multimedia presentation is intended to be presented at a first frame rate;

means for decoding visual data from the encoded source data;

said means including converting said visual data if it is in a progressive format into an interlaced format, for presentation on an interlaced format display device, and converting said visual data if it is in an interlaced format into a progressive format, for presentation on a progressive format display device;

means for decoding audio data from the encoded source data;

means for performing a time-stretching process on the decoded audio data, where the time stretching process changes a number of samples of the audio data such that the audio data can be presented at a second effective frame rate that is effectively different from the first frame rate, where a pitch of the audio data is controlled;

and

means for providing decoded visual data and decoded audio data for presentation at the second effective frame rate, where the decoded visual data and the decoded audio data are synchronized.

68. A digital video player that embodies the circuit of claim 67.

69. A digital video recorder (DVR) that embodies the circuit of claim 67.

70. A DVD player that embodies the circuit of claim 67.

71. A method of presenting an original source multimedia presentation comprising:

receiving an indication of a selected time duration;

converting the original source multimedia presentation if it is in a progressive format into an interlaced format, for presentation on an interlaced format display device, and converting the original source multimedia presentation if it is in an interlaced format into a progressive format, for presentation on a progressive format display device;

playing back the original source multimedia presentation at a revised playback rate from an original playback rate such that playback can be completed within the selected time duration; and

modifying audio data during playback such that a pitch of a time-stretched audio data is controlled.

72. The method as defined in claim 71, wherein modifying audio data further comprises modifying such that a pitch of the time-stretched audio data sounds unchanged from the original playback rate to the revised playback rate.

73. The method as defined in claim 71, further comprising providing a user interface configured to permit entry of the selected time duration.

74. A method of presenting an original source multimedia presentation comprising:

receiving an indication of a selected time;

converting the original source multimedia presentation if it is in a progressive format into an interlaced format, for presentation on an interlaced format display device, and converting the original source multimedia presentation if it is in an interlaced format into a progressive format, for presentation on a progressive format display device;

playing back the original source multimedia presentation at a revised playback rate from an original playback rate such that playback can be completed by the selected time; and

modifying audio data during playback such that a pitch of a time-stretched audio data is controlled.

75. The method as defined in claim 74, wherein modifying audio data further comprises modifying such that a pitch of the time-stretched audio data sounds unchanged from the original playback rate to the revised playback rate.

76. The method as defined in claim 74, further comprising providing a user interface configured to permit entry of the selected time.

77. The method as defined in claim 74, further comprising:

determining whether playback can be completed by the selected time at the original playback rate; and

not changing to a revised playback rate when playback can be completed by the selected time at the original playback rate.

78. A computer readable storage medium, encoded with a computer program and user interface comprising:

a module configured to receive an indication of a request for a speed of an original source multimedia presentation where the speed relates to a playback speed and can correspond to playback speeds other than a normal playback speed;

wherein if said original source multimedia presentation is in with a progressive format it is first converted into an interlaced format, for presentation on an interlaced format display device, and if said original source multimedia presentation is in with an interlaced format it is first converted into a progressive format, for presentation on a progressive format display device;

a module configured to provide the indication to a video decoder

a module configured to modify audio data to a time stretched audio data such that a pitch of the time stretched audio data sounds unchanged at the playback speed.

79. The user interface as defined in claim 78, wherein the request for the change in speed is expressed relative to an existing speed.

80. The user interface as defined in claim 78, wherein the indication corresponds to a time duration, where the time duration corresponds to when completed playback of the original source multimedia presentation is desired.

81. The user interface as defined in claim 78, wherein the indication corresponds to a selected time, which corresponds to when completed playback of the digital original source multimedia presentation is desired.

82. A method of presenting a digital original source multimedia presentation, where the method comprises:

presenting a source of visual data and audio data at a playback rate other than a speed of playback of the digital original source multimedia presentation as specified by a corresponding video compression standard;

converting the original source multimedia presentation if it is in a progressive format into an interlaced format, for presentation on an interlaced format display device, and converting the digital original source multimedia presentation if it is in an interlaced format into a progressive format, for presentation on a progressive format display device; and

modifying the audio data to a time-stretched audio data such that a pitch of the time-stretched audio data is controlled.

83. The method as defined in claim 82, wherein modifying the audio data further comprises modifying such that the pitch of the time-stretched audio data sounds unchanged at the playback rate.

84. The method as defined in claim 82, wherein the method is performed in real time.

85. The method as defined in claim 82, wherein the video compression standard corresponds to MPEG-2.

86. The method as defined in claim 82, further comprising changing the playback rate to another playback rate in real time while the method is performed.

87. The method as defined in claim 82, further comprising:

receiving an indication of a selected amount of time; and

selecting the playback rate such that playback can be completed within the selected time.

88. The method as defined in claim 82, further comprising:

receiving an indication of a selected time; and

selecting the playback rate such that playback can be completed by the selected time.

89. The method as defined in claim 82, further comprising providing a user interface for setting the playback rate.

90. A computer program product containing a computer readable storage medium with computer executable instructions embodied therein for presenting a digital original multimedia presentation, comprising:

instructions for presenting visual data and audio data at a playback rate other than a speed of playback of the digital original multimedia presentation as specified by a corresponding video compression standard;

wherein the instructions for presenting the visual data include instructions for converting said visual data if it is in a progressive format into an interlaced format, for presentation on an interlaced format display device, and instructions for converting said visual data if it is in an interlaced format into a progressive format, for presentation on a progressive format display device; and

instructions for modifying the audio data to a time-stretched audio data such that a pitch of the time-stretched audio data is controlled.

91. A circuit for presenting a digital original multimedia presentation in a video decoder comprising:

means for presenting visual data and audio data at a playback rate other than a speed of playback of the digital original multimedia presentation as specified by a corresponding video compression standard;

wherein the means for presenting visual data include means for converting said visual data if it is in a progressive format into an interlaced format, for presentation on an interlaced format display device, and means for converting said visual data if it is in an interlaced format into a progressive format, for presentation on a progressive format display device; and

means for modifying the audio data to a time-stretched audio data such that a pitch of the time-stretched audio data is controlled.

93. A digital video player that embodies the circuit of claim 91.

94. A DVD player that embodies the circuit of claim 91.

95. A digital video recorder (DVR) that embodies the circuit of claim 91.

92. A circuit for presenting a digital original multimedia presentation in a video decoder comprising:

a module configured to present visual data and audio data at a playback rate other than a speed of playback of the digital original multimedia presentation as specified by a corresponding video compression standard;

wherein the presentation of the visual data includes converting said visual data if it is in a progressive format into an interlaced format, for presentation on an interlaced format display device, and converting said visual data if it is in an interlaced format into a progressive format, for presentation on a progressive format display device; and

a module configured to modify the audio data to a time-stretched audio data such that a pitch of the time-stretched audio data sounds unchanged at the playback rate.